This module is designed to provide you an introduction to the keras
API, deep learning and some of the key components that make DL
algorithms run. Throughout this module you will learn about:
- Perceptrons
- Gradient descent
- Activation functions
- Learning rate & momentum
- Model capacity (width vs.Ā depth)
- Learning curves
- Batch size
Package Requirements
Loading
Letās load the keras package along with a couple other
packages weāll use.
library(keras) # for modeling
library(tidyverse) # for wrangling & visualization
library(glue) # for string literals
Installation
Normally, you will be starting out from scratch and need to install
and set up keras on your own laptop. First, you need to install keras
from CRAN. Once the package is installed, you need to install the Keras
and TensorFlow Python packages, which is what the R Keras and TensorFlow
packages communicate with. keras simplifies this with
install_keras() which allows for:
- both GPU & CPU options setups
- installation in a virtual or conda environment
- setup for Theano & CNTK backends rather than TensorFlow
See https://keras.rstudio.com/ for details.
# install keras if necessary
# install.packages("keras")
# default CPU-based installations of Keras and TensorFlow
# keras::install_keras()
# for GPU installation
# install_keras(tensorflow = "gpu")
For this workshop we will be using a cloud environment to ensure we
are all operating in common environment and Keras and TensorFlow have
already been installed.
Simple Linear Regression
For our first example, letās look at a very simple linear problem
based on data with:
- obs = 1,000
- intercept = 30
- slope = 5
- with a little bit of random noise
n <- 1000 # n observations
b <- 30 # intercept
a <- 5 # slope
set.seed(123)
(df <- tibble(
x = runif(n, min = -1, max = 1),
y = b + a*x + rnorm(n)
))
Simple regression with OLS
Iām assuming weāre all familiar with the OLS methodology for linear
regression modeling where our objective function (aka
loss) is:
\[loss = MSE = \frac{1}{n}\sum^n_{i-1}(Y_i
- \hat Y_i)^2\]
With OLS, we can find the minimum MSE with a closed form solution
(donāt worry, this equation is not important š¬):
\[ \hat \beta =
(X^TX)^{-1}X^Ty\]
which will provide us with our coefficients (\(b\)) in the equation:
\[\hat y = b_0 + b_1x\]
If we apply OLS to our data, we get:
- estimated intercept = 30.01
- estimated slope = 4.97
- loss score (MSE) = 1.002
lm_model <- summary(lm(y ~ x, data = df))
lm_model
We can illustrate this model fit to our data:
mse <- lm_model[["sigma"]]
ggplot(df, aes(x, y)) +
geom_point(alpha = 0.5) +
geom_smooth(method = "lm", se = FALSE) +
ggtitle(glue("MSE = {mse}"))
Simple regression with a perceptron
Now, letās illustrate performing a similar process but with a basic
building block of neural networks ā the perceptron.
To model with keras we need our data to be in
tensors. Weāll discuss tensors more later but
for now just realize that:
- 1D tensor = vector
- 2D tensor = matrix
x <- as.matrix(df$x)
y <- df$y
Training a neural network model consists of 3 steps:
- Define model architecture
- Define how our model is going to learn
- Train our model
1. Define model architecture
Defining an architecture includes defining the type of model and the
arrangement of layers:
model <- keras_model_sequential() %>%
layer_dense(units = 1, input_shape = ncol(x))
- Define structure of model
- Sequential: linear layers (i.e.Ā MLP, CNN, RNN, LSTM)
- Functional: adds more flexibility (i.e.Ā combining CNN & LSTM to
predict probability of cancer)
- Define arrangement and shape of layers
layer_dense: a single layer of nodes
units = 1: a single perceptron unit in our layer
input_shape: we need to tell our first layer how many
inputs to expect

All this is calculating is (if we match the above image):
\[y = bias + w_1x_1\]
2. Define how our model is going to learn
The whole goal of training a neural network is to find the optimal
set of parameter weights (aka coefficients in OLS-speak).
Our goal is to find weights that minimize the loss
score
We define how our model is going to learn with
compile():
model %>% compile(
optimizer = "sgd",
loss = "mse"
)
Optimizer: neural networks learn via
backpropagation. There are various
backpropagation algorithms but weāll start
with the most basic⦠stochastic gradient descent (SGD).
loss: how do we want to measure our modelās
error. keras comes with many built-in loss functions and we can even
create custom loss functions. Here, weāll use MSE.

3. Train our model
The last thing we need is how to train our model, which we do with
fit():
history <- model %>% fit(x, y, batch_size = 32, epochs = 10)
x: feature tensor
y: target tensor
batch_size: pick observations from our training data,
perform forward pass, compute loss score, compute gradient, perform
backward pass, update our weight (default = 32).
epoch: 1 epoch = one forward pass and one backward pass
of all the training examples. Weāre repeating that 10 times (default =
10).
Putting it all together
Letās put all three steps together and train our model. Hereās a
visual depiction:

And hereās the code:
# 1. Define model architecture
model <- keras_model_sequential() %>%
layer_dense(units = 1, input_shape = ncol(x))
# 2. Define how our model is going to learn
model %>% compile(
optimizer = "sgd",
loss = "mse"
)
# 3. Train our model
history <- model %>% fit(x, y, epochs = 10)
Whereas in OLS we call the intercept and slope parameters
ācoefficientsā, in neural networks we call them
weights. We can see that after 10 epochs our
weights are getting close to the underlying ātruthā values (slope = 5,
intercept = 30) but also notice that our MSE loss score is still
decreasing after 10 epochs.
get_weights(model)
Models are object oriented
Note that modeling with keras/tensorflow in R may feel a bit
different than other modeling packages youāve used in R. Since
keras/tensorflow are reticulated from Python, the model is an
object oriented object with Python
attributes.
- Object oriented - our model object changes without assignment:
model %>% compile() changed our model object by
adding the optimizer and loss parameter arguments
model %>% fit() will continue to build onto our
existing model
# let's execute one more epoch. Note how our loss decreases from the last epoch
# above
model %>% fit(x, y, epoch = 1)
# our model weights get updated from this last epoch and continue to get closer
# to the underlying true values
get_weights(model)
- Our model is a Python object - there will be things you can not
directly access because they are Python objects. However, for most
things that you want to access there will be a function to export
them:
# the model weights are held in Python numpy arrays
model$weights
# we use helper functions to export these kinds of objects
get_weights(model)
Your Turn (3 min)
- Fill in the blanks below and train the model for 25 epochs.
- Explore the
history object.
- What are the final weights for this model? How do they compare to
the underlying intercept (30) and slope (5)?
# 1. Define model architecture
model <- keras_model_sequential() %>%
layer_dense(units = ___, input_shape = ____)
# 2. Define how our model is going to learn
model %>% compile(
optimizer = "sgd",
loss = ____
)
# 3. Train our model
history <- model %>% fit(x, y, epochs = ____)
Gradient descent
We can see the progression of the gradient descent process by
examining the weights our model produces after each epoch. This is not
something you will do often but, rather, helps make the gradient descent
process more concrete.
# data frame to dump our results
model_est <- expand.grid(
epoch = 1:25,
a_sgd = NA,
b_sgd = NA
)
# 1. Define model architecture
model <- keras_model_sequential() %>%
layer_dense(units = 1, input_shape = ncol(x))
# 2. Define how our model is going to learn
model %>% compile(
optimizer = "sgd",
loss = "mse"
)
# 3. Train our model for 25 epochs and record the weights after each epoch
for (row in seq_len(nrow(model_est))) {
history <- model %>% fit(x, y, epoch = 1, verbose = FALSE)
current_wts <- get_weights(model)
model_est[row, "a_sgd"] <- current_wts[[1]]
model_est[row, "b_sgd"] <- current_wts[[2]]
}
The following table shows the estimate slope (a_sgd) and
intercept (b_sgd) produced by SGD after each epoch.
model_est
history
We can visualize our linear model after each epoch with the
following. Note how each epoch results in a linear prediction (dotted)
that gets closer to the truth (blue line). After 25 epochs our model
basically converges to the same results (yellow dotted line).
We can see this with our loss (MSE) that nearly equates the OLS MSE
(1.00187).
epoch_pred <- merge(df, model_est, all = TRUE) %>%
mutate(pred = b_sgd + a_sgd*x)
last_epoch <- filter(epoch_pred, epoch == max(epoch))
ggplot(data = df, aes(x, y)) +
geom_point(alpha = 0.1) +
geom_smooth(method = "lm", se = FALSE, size = 1.5) +
geom_line(data = epoch_pred, aes(x, pred, group = epoch), lty = "dotted") +
geom_line(data = last_epoch, aes(x, pred, group = epoch),
lty = "dotted", color = "yellow", size = 2) +
ggtitle(glue("MSE = {history$metrics$loss}"))
Key Takeaways
- A basic single perceptron computes the same transformation as
OLS:
\[\hat y = bias + weight_1 \times x_1 +
weight_2 \times x_2 + \dots + weight_n \times x_n\]
- Neural networks learn via gradient descent - an iterative approach
of predicting with a forward pass, measuring the gradient of the error,
and performing a backward pass to update the weights based on the
gradient.
Binary Classification
Letās do the same process but now weāll do so with a binary
classification problem (i.e.Ā predicting yes vs.Ā no response).
set.seed(123)
generated <- mlbench::mlbench.simplex(n = 1000, d = 1, sd = .3)
x <- generated$x
y <- ifelse(generated$classes == 1, 0, 1)
(df <- tibble(x = as.vector(x), y = y))
Our generated data has some overlap so there is no linear seperation
without having some error. Note than when discussing binary
classification problems, we will mainly use the crossentropy (aka log
loss) loss function.
glm_model <- glm(y ~ x, family = binomial(link = "logit"), data = df)
crossentropy <- MLmetrics::LogLoss(glm_model$fitted.values, df$y)
ggplot(df, aes(x, y)) +
geom_point(aes(color = as.factor(y)), size = 2, show.legend = FALSE) +
geom_smooth(method = "glm", method.args = list(family = "binomial"), se = FALSE) +
ggtitle(glue("crossentropy = {crossentropy}")) +
ylab("probability y = 1")
Sigmoid Activation Function
When predicting a binary response, we typically want to predict a
real value between 0-1 representing the probability of the positive
binary class. Unfortunately our regular perceptron creates a linear
transformation. However, we can apply an activation
function to transform this linear transformation to a
non-linear transformation.
When predicting a binary response, we use a
sigmoid activation to convert our linear
transformation to a 0-1 probability of the positive class.
\[sigmoid(y) =
\frac{1}{1+e^{-y}}\]

When predicting a binary response, we need to make the following
changes to our code:
- Add
activation = "sigmoid" to the layer that is
predicting the output.
- Note that since we are predicting the probability from 0-1 for our
response, we keep
units = 1.
- loss - we change
loss = "binary_crossentropy" to use
the crossentropy / log loss objective function.
model <- keras_model_sequential() %>%
layer_dense(units = 1, input_shape = ncol(x), activation = "sigmoid")
model %>% compile(
optimizer = "sgd",
loss = "binary_crossentropy"
)
(history <- model %>% fit(x, y, epochs = 50, verbose = FALSE))
We see that our loss is quite a bit off from our logistic regression
model.
df %>%
mutate(pred = predict(model, x) %>% as.vector()) %>%
ggplot(aes(x, y)) +
geom_point(aes(color = as.factor(y)), size = 2, show.legend = FALSE) +
geom_smooth(method = "glm", method.args = list(family = "binomial"), se = FALSE) +
geom_line(aes(y = pred), lty = "dashed") +
ggtitle(glue("crossentropy = {min(history$metrics$loss)}")) +
ylab("probability of y = 1")
However, if we look at our loss scores, we see that they are still
improving, its just taking a long time. Plus, it looks like there is
more improvement that can be made to our loss.
plot(history)
Learning rate and momentum
An important parameter in gradient descent is the size of the steps
which is controlled by the learning rate. If
the learning rate isā¦
- too small: the algorithm will take many iterations (steps) to find
the minimum
- too large: you might jump across the minimum and end up further away
than when you started

The default learning rate for SGD is 0.01. Unfortunately with this
rate, it will take over 1,000 epochs to reach a loss score comparable to
logistic regression. However, we can customize our optimizer with
optimizer_sdg():
model <- keras_model_sequential() %>%
layer_dense(units = 1, input_shape = ncol(x), activation = "sigmoid")
model %>% compile(
optimizer = optimizer_sgd(lr = 0.1),
loss = "binary_crossentropy"
)
(history <- model %>% fit(x, y, epochs = 50, verbose = FALSE))
Another common approach to adjust our learning rate is to add
momentum. Adding momentum allows our learning
rate to adapt. Momentum simply adds a fraction of the previous weight
update to the current one.

Letās add some momentum to our learning rate. We see that our loss
improves even more within the same number of epochs.
model <- keras_model_sequential() %>%
layer_dense(units = 1, input_shape = ncol(x), activation = "sigmoid")
model %>% compile(
optimizer = optimizer_sgd(lr = 0.1, momentum = 0.5),
loss = "binary_crossentropy"
)
(history <- model %>% fit(x, y, epochs = 50, verbose = FALSE))
Your Turn (3 min)
- Try different combinations of learning rate and momentum. A few
rules of š:
- Typically, we start assessing learning rates in log values ranges of
[1e-1, 1e-7] (i.e.Ā 0.1, 0.01, ā¦, 0.0000001).
- Momentum is typically > 0.5 and often in the 0.9-0.99 range.
- Plot the loss learning curve.
- How does your final loss compare to logistic regression?
model <- keras_model_sequential() %>%
layer_dense(units = 1, input_shape = ncol(x), activation = "sigmoid")
model %>% compile(
optimizer = optimizer_sgd(lr = 0.1, momentum = .95),
loss = "binary_crossentropy"
)
(history <- model %>% fit(x, y, epochs = 50, verbose = FALSE))
Key takeaways
- Activation functions:
- We use activation functions to transform the perceptronās linear
equation to a non-linear form.
- For binary classification problems we use the āsigmoidā activation
to convert our predictions to a 0-1 probability.
- Learning rate:
- We can control the rate of learning by increasing & decreasing
the learning rate.
- We can make this learning rate adaptive to the curvature of our loss
gradient by incorporating momentum.
Non-linear Patterns
As our datasets get larger or include non-linearities, our model
needs to become more sophisticated. For this example, weāll stick with
one predictor variable but weāll add a non-linearity component:
set.seed(123)
df <- tibble(
x = seq(from = -1, to = 2 * pi, length = n),
e = rnorm(n, sd = 0.2),
y = sin(x) + e
)
ggplot(df, aes(x, y)) +
geom_point(alpha = 0.5) +
geom_smooth(se = FALSE)
Again, letās extract our feature and target tensors:
x <- as.matrix(df$x)
y <- df$y
As our underlying model has more complexity, we add hidden layers to
capture non-linearities and interactions. We call these neural network
models multi-layer perceptrons (MLPs); also
referred to as densely connected feed forward
networks.

We can add a hidden layers by adding additional
layer_dense() functions to our model architecture. For
example, the following code would create an MLP with:
- 3 hidden layers:
- each hidden layer has 16 nodes
- only the first hidden layer requires
input_shape
- each hidden layer uses a ReLU activation function (weāll discuss
shortly)
- the last
layer_dense() is always the output layer
- activation function for output layer is always dependent on the
problem
- regression: NULL
- binary classification:
activation = "signmoid"
- multi-class classification:
activation = "softmax"
model <- keras_model_sequential() %>%
layer_dense(units = 16, input_shape = ncol(x), activation = "relu") %>%
layer_dense(units = 16, activation = "relu") %>%
layer_dense(units = 16, activation = "relu") %>%
layer_dense(units = 1)
Why ReLU
The rectified linear activation function is very simple; if the
linear transformation within the perceptron results in a negative number
then the output is 0. If its positive then its that value.
\[ReLU = max(0, z)\]

Benefits (see http://proceedings.mlr.press/v15/glorot11a/glorot11a.pdf):
- Simple geometric transformations can produce very complex
patterns.
- Computational simplicity (easy to compute the gradient)
- Representational sparcity (forcing 0s results in sparse
outputs)
- Linearity (reduces vanishing gradient descent - discussed
later)

Letās see this in action:
# library(reticulate)
# library(keras)
# use_condaenv("r-tensorflow")
model <- keras_model_sequential() %>%
layer_dense(units = 16, input_shape = ncol(x), activation = "relu") %>%
layer_dense(units = 1)
model %>% compile(
optimizer = optimizer_sgd(lr = 0.01, momentum = .9),
loss = "mse"
)
(history <- model %>% fit(x, y, epochs = 50, verbose = FALSE))
df %>%
mutate(pred = predict(model, x) %>% as.vector()) %>%
ggplot(aes(x, y)) +
geom_point(alpha = 0.05) +
geom_smooth(se = FALSE) +
geom_line(aes(y = pred), lty = "dashed", color = "red", size = 1)
Model capacity
Model capacity determines the extend to
which our model can capture underlying relationships and patterns. We
control model capacity with:
- width: number of units in a layer
- Rule of š: typically use powers of 2 (i.e.Ā 16, 32, 64, 128, 256,
512)
- depth: number of hidden layers
- Rule of š: we often see better performance (accuracy & compute
efficiency) by increasing the number of layers moreso than nodes.
Letās add 2 hidden layers, each with 16 units:
model <- keras_model_sequential() %>%
layer_dense(units = 16, input_shape = ncol(x), activation = "relu") %>%
layer_dense(units = 16, activation = "relu") %>%
layer_dense(units = 1)
model %>% compile(
optimizer = optimizer_sgd(lr = 0.01, momentum = .9),
loss = "mse"
)
(history <- model %>% fit(x, y, epochs = 50, verbose = FALSE))
Looking at how our predicted values fit the true underlying
model:
df %>%
mutate(pred = predict(model, x) %>% as.vector()) %>%
ggplot(aes(x, y)) +
geom_point(alpha = 0.05) +
geom_smooth(se = FALSE) +
geom_line(aes(y = pred), lty = "dashed", color = "red", size = 1)
Your Turn (3 min)
- Try using only one hidden layer and increase the width to 32, 64,
128, 256
- Try adding a second layer and increase the width of each layer
progressively
- Rule of š: when we add more layers we typically have the following
patterns:
- tunnel shaped: each hidden layer has the same number of units
- funnel shaped: hidden layers progressively get smaller
model <- keras_model_sequential() %>%
layer_dense(units = ____, input_shape = ____, activation = ____) %>%
layer_dense(units = 1)
model %>% compile(
optimizer = optimizer_sgd(lr = 0.01, momentum = .9),
loss = "mse"
)
(history <- model %>% fit(x, y, epochs = 50, verbose = FALSE))
Run the following to see how your adjusted model fits the actual
data:
df %>%
mutate(pred = predict(model, x) %>% as.vector()) %>%
ggplot(aes(x, y)) +
geom_point(alpha = 0.25) +
geom_smooth(se = FALSE) +
geom_line(aes(y = pred), lty = "dashed", color = "red", size = 1)
Key Takeaways
- Hidden layers almost always use the ReLU activation function (this
should be your default).
- Control model capacity by width and depth (adding depth typically
outperforms simply focusing on width).
Multi-predictor Multi-class Classification
Letās get a little more complicated now and look at a dataset that
has:
- 3 predictor variables
- 4 response classes
set.seed(123)
generated <- mlbench::mlbench.simplex(n = n*2, d = 3, sd = 0.3)
plot(generated)
# 3 features
head(generated$x)
# 4 response categories
head(generated$classes)
Preparing our data takes a little more effort in this case:
- features: our features are already a matrix so
weāre good
- response: our response is a factor which we are
going to convert to a matrix:
to_categorical dummy encodes our classes. This allows
us to compute the predicted probability for each class
to_categorical expects a zero-based input from 0-n
(Python š)
x <- generated$x
y <- generated$classes %>% as.numeric()
y <- to_categorical(y - 1)
n_classes <- ncol(y)
# our preprocesses response
head(y)
Fit model using validation
In practice we are unable to visualize the fit of our data to
understand variance-bias tradeoff (i.e.Ā are we over or underfitting our
data). Consequently, we rely on using a validation set and what we call
learning curves.
- validation_split: will train model on first 80% of
data and use the last 20% of data to see assess performance.
- metrics: often we want to assess alternative
metrics along with our loss score.
model <- keras_model_sequential() %>%
layer_dense(units = 16, input_shape = ncol(x), activation = "relu") %>%
layer_dense(units = n_classes, activation = "softmax")
model %>% compile(
optimizer = optimizer_sgd(lr = 0.01, momentum = .9),
loss = "categorical_crossentropy",
metrics = "accuracy"
)
history <- model %>% fit(
x, y,
batch_size = 32,
epochs = 20,
validation_split = 0.2
)
Our learning curve shows some unique behavior. We can learn a lot
about our model by paying attention to learning curves (see this extra
notebook: https://misk-data-science.github.io/misk-dl/notebooks/99x4-learning-curve-diagnostics.nb.html)
history
plot(history)
In this case, the problem is that our data is ordered so the last 20%
of our data contains only one class. So we always want to make sure we
are randomizing our data.
set.seed(123)
randomize <- sample(seq_len(n), size = n, replace = FALSE)
x <- x[randomize, ]
y <- y[randomize, ]
Now letās try the same model again.
model <- keras_model_sequential() %>%
layer_dense(units = 16, input_shape = ncol(x), activation = "relu") %>%
layer_dense(units = n_classes, activation = "softmax")
model %>% compile(
optimizer = optimizer_sgd(lr = 0.01, momentum = .9),
loss = "categorical_crossentropy",
metrics = "accuracy"
)
history <- model %>% fit(
x, y,
batch_size = 32,
epochs = 20,
validation_split = 0.2
)
Our results look much better. Our loss curve shows a few things that
we always want to strive for:
- The training and validation loss curves are very close to one
another. Typically, there will be a gap between the two but our goal
should be to minimize this gap. This leads to a more stable model that
generalizes better
- We prefer that our validation loss is above the training loss (when
our metric is designed for ālower-is-betterā). If our validation loss
below (better) then the training loss then that typically means we are
underfitting and we should increase capacity.
- Our validation loss has stopped improving, which means we have
trained it for enough epochs.
- We want our validation loss to be as smooth as possible (iratic
behavior means unstable generalization).
history
plot(history)
Effects of batch size
So far weāve just been using the default batch size of 32. However,
there are other options:
- Batch gradient descent: computes the derivative of
the gradient based on the entire dataset (all observations).
- provides more accurate and smooth gradient descent butā¦
- scales horribly to large data
- Stochastic gradient descent: randomly selects an
individual observations, computes gradients and updates model weights
after this single observation has been evaluated.
- provides quick feedback so the model learns quickly andā¦
- results in noisy gradient descent which helps avoid local minimums
butā¦
- noisy gradient descent makes it hard to converge on global minimum
andā¦
- can result in unstable generalization
- Mini-batch gradient descent: randomly selects a
subset of observations, computes gradients and updates model weights
after this subset has been evaluated.
- Balances efficiencies of batch vs.Ā stochastic
- Balances robust convergence of batch with some stochastic nature to
minimize local minima.
- But one more hyperparameter to think about.
- Most common: \(2^s\): 32, 64, 128,
256, 512
Go ahead and try:
batch_size = 1 (stochastic gradient descent)
batch_size = nrow(x) (batch gradient descent)
batch_size = b where b equals 16, 32, 64,
128
Note: batch size and learning rate often interact
and should be tuned together.
model <- keras_model_sequential() %>%
layer_dense(units = 16, input_shape = ncol(x), activation = "relu") %>%
layer_dense(units = n_classes, activation = "softmax")
model %>% compile(
optimizer = optimizer_sgd(lr = 0.01, momentum = .9),
loss = "categorical_crossentropy",
metrics = "accuracy"
)
history <- model %>% fit(
x, y,
batch_size = ____,
epochs = 20,
validation_split = 0.2
)
Making predictions
When predicting with classification models we can either predict the
class (based on probability > 0.5) or predict the probabilities for
each class:
# predicting probabilities
model %>% predict(x) %>% head()
# predicting classes
model %>% predict_classes(x) %>% head()
Key Takeaways
- Monitor the learning curves to diagnose model performance.
- Batch size effects our learning curve. Mini-batch sizes of 32, 64,
128, 256 tend to perform best.
Mini-Project (time dependent)
Time to work with some real (although unexciting) data -
iris. This data contains:
- 4 features (
Sepal.Length, Sepal.Width,
Petal.Length, Petal.Width)
- 3 response classes (
Species = setosa, versicolor,
verginica)
head(iris)
Data prep
First step is to prepare our data. This involves converting our
features to a tensor (aka matrix). Also, since our response variable is
multi-class, we want to convert it to a 2D tensor (aka matrix). We also
need to randomize the data.
These steps are provided for you:
# convert features to a tensor (aka matrix)
x <- iris[1:4] %>% as.matrix()
# convert response to a multi-class tensor (aka matrix)
y <- iris$Species %>% as.numeric()
y <- to_categorical(y - 1)
# randomize data
set.seed(123)
total_obs <- nrow(x)
randomize <- sample(seq_len(total_obs), size = total_obs, replace = TRUE)
x <- x[randomize, ]
y <- y[randomize, ]
Note that our response tensor (y) has 3 columns. These
columns relate alphabetically to our response classes:
- column 1 = setosa
- column 2 = versicolor
- column 3 = virginica
head(y)
Modeling
Start with the following:
- 1 hidden layer with 16 units
- learning rate of 0.01 and no momentum
- 20 epochs with batch sizes of 32
- validation split of 20%
Then start adjusting the following:
- learning rate (maybe add momentum)
- model capacity (try wider and/or deeper capacity)
- batch size (do larger or smaller batch sizes help performance)
- epochs (do you need more or less epochs to reach a minimum
validation loss)
# define architecture
model <- keras_model_sequential() %>%
layer_dense(units = ____, input_shape = ____, activation = ____) %>%
layer_dense(units = ____, activation = ____)
# define learning procedure
model %>% compile(
optimizer = optimizer_sgd(lr = ____),
loss = "categorical_crossentropy",
metrics = "accuracy"
)
# train model
history <- model %>% fit(
x, y,
batch_size = ____,
epochs = ____,
validation_split = ____
)
Summary
This module is meant to only introduce some of the key ingredients
involved in training a basic MLP model. However, as the mini-project
probably demonstrated, it didnāt do much to help you understand how to
put these ingredients together in a methodolical approach to maximize
model performance. The next module aims to fill this gap and provide
some best practices for training a model.
LS0tCnRpdGxlOiAiTWFpbiBJbmdyZWRpZW50cyIKb3V0cHV0OgogIGh0bWxfZG9jdW1lbnQ6CiAgICB0b2M6IHllcwogICAgZGZfcHJpbnQ6IHBhZ2VkCiAgaHRtbF9ub3RlYm9vazoKICAgIHRvYzogeWVzCiAgICB0b2NfZmxvYXQ6IHllcwotLS0KCmBgYHtyIHNldHVwLCBpbmNsdWRlPUZBTFNFfQprbml0cjo6b3B0c19jaHVuayRzZXQoZWNobyA9IFRSVUUsIG1lc3NhZ2UgPSBGQUxTRSwgd2FybmluZyA9IEZBTFNFKQpnZ3Bsb3QyOjp0aGVtZV9zZXQoZ2dwbG90Mjo6dGhlbWVfbWluaW1hbCgpKQpgYGAKClRoaXMgbW9kdWxlIGlzIGRlc2lnbmVkIHRvIHByb3ZpZGUgeW91IGFuIGludHJvZHVjdGlvbiB0byB0aGUga2VyYXMgQVBJLCBkZWVwCmxlYXJuaW5nIGFuZCBzb21lIG9mIHRoZSBrZXkgY29tcG9uZW50cyB0aGF0IG1ha2UgREwgYWxnb3JpdGhtcyBydW4uIFRocm91Z2hvdXQKdGhpcyBtb2R1bGUgeW91IHdpbGwgbGVhcm4gYWJvdXQ6CgoqIFBlcmNlcHRyb25zCiogR3JhZGllbnQgZGVzY2VudAoqIEFjdGl2YXRpb24gZnVuY3Rpb25zCiogTGVhcm5pbmcgcmF0ZSAmIG1vbWVudHVtCiogTW9kZWwgY2FwYWNpdHkgKHdpZHRoIHZzLiBkZXB0aCkKKiBMZWFybmluZyBjdXJ2ZXMKKiBCYXRjaCBzaXplCgojIFBhY2thZ2UgUmVxdWlyZW1lbnRzIHsudGFic2V0IC50YWJzZXQtZmFkZX0KCiMjIExvYWRpbmcKCkxldCdzIGxvYWQgdGhlIGBrZXJhc2AgcGFja2FnZSBhbG9uZyB3aXRoIGEgY291cGxlIG90aGVyIHBhY2thZ2VzIHdlJ2xsIHVzZS4KCmBgYHtyfQpsaWJyYXJ5KGtlcmFzKSAgICAgICAjIGZvciBtb2RlbGluZwpsaWJyYXJ5KHRpZHl2ZXJzZSkgICAjIGZvciB3cmFuZ2xpbmcgJiB2aXN1YWxpemF0aW9uCmxpYnJhcnkoZ2x1ZSkgICAgICAgICMgZm9yIHN0cmluZyBsaXRlcmFscwpgYGAKCiMjIEluc3RhbGxhdGlvbgoKTm9ybWFsbHksIHlvdSB3aWxsIGJlIHN0YXJ0aW5nIG91dCBmcm9tIHNjcmF0Y2ggYW5kIG5lZWQgdG8gaW5zdGFsbCBhbmQgc2V0IHVwCmtlcmFzIG9uIHlvdXIgb3duIGxhcHRvcC4gRmlyc3QsIHlvdSBuZWVkIHRvIGluc3RhbGwga2VyYXMgZnJvbSBDUkFOLiBPbmNlIHRoZQpwYWNrYWdlIGlzIGluc3RhbGxlZCwgeW91IG5lZWQgdG8gaW5zdGFsbCB0aGUgS2VyYXMgYW5kIFRlbnNvckZsb3cgUHl0aG9uCnBhY2thZ2VzLCB3aGljaCBpcyB3aGF0IHRoZSBSIEtlcmFzIGFuZCBUZW5zb3JGbG93IHBhY2thZ2VzIGNvbW11bmljYXRlCndpdGguIGtlcmFzIHNpbXBsaWZpZXMgdGhpcyB3aXRoIGBpbnN0YWxsX2tlcmFzKClgIHdoaWNoIGFsbG93cyBmb3I6CgoqIGJvdGggR1BVICYgQ1BVIG9wdGlvbnMgc2V0dXBzCiogaW5zdGFsbGF0aW9uIGluIGEgdmlydHVhbCBvciBjb25kYSBlbnZpcm9ubWVudAoqIHNldHVwIGZvciBUaGVhbm8gJiBDTlRLIGJhY2tlbmRzIHJhdGhlciB0aGFuIFRlbnNvckZsb3cKClNlZSBodHRwczovL2tlcmFzLnJzdHVkaW8uY29tLyBmb3IgZGV0YWlscy4KCmBgYHtyIGluc3RhbGwsIGV2YWwgPSBGQUxTRX0KIyBpbnN0YWxsIGtlcmFzIGlmIG5lY2Vzc2FyeQojIGluc3RhbGwucGFja2FnZXMoImtlcmFzIikKCiMgZGVmYXVsdCBDUFUtYmFzZWQgaW5zdGFsbGF0aW9ucyBvZiBLZXJhcyBhbmQgVGVuc29yRmxvdwojIGtlcmFzOjppbnN0YWxsX2tlcmFzKCkKCiMgZm9yIEdQVSBpbnN0YWxsYXRpb24KIyBpbnN0YWxsX2tlcmFzKHRlbnNvcmZsb3cgPSAiZ3B1IikKYGBgCgpGb3IgdGhpcyB3b3Jrc2hvcCB3ZSB3aWxsIGJlIHVzaW5nIGEgY2xvdWQgZW52aXJvbm1lbnQgdG8gZW5zdXJlIHdlIGFyZSBhbGwgCm9wZXJhdGluZyBpbiBjb21tb24gZW52aXJvbm1lbnQgYW5kIEtlcmFzIGFuZCBUZW5zb3JGbG93IGhhdmUgYWxyZWFkeSBiZWVuIAppbnN0YWxsZWQuCgojIFNpbXBsZSBMaW5lYXIgUmVncmVzc2lvbgoKRm9yIG91ciBmaXJzdCBleGFtcGxlLCBsZXQncyBsb29rIGF0IGEgdmVyeSBzaW1wbGUgbGluZWFyIHByb2JsZW0gYmFzZWQgb24gZGF0YQp3aXRoOgoKKiBvYnMgPSAxLDAwMAoqIGludGVyY2VwdCA9IDMwCiogc2xvcGUgPSA1Ciogd2l0aCBhIGxpdHRsZSBiaXQgb2YgcmFuZG9tIG5vaXNlCgpgYGB7cn0KbiA8LSAxMDAwICAgIyBuIG9ic2VydmF0aW9ucwpiIDwtIDMwICAgICAjIGludGVyY2VwdAphIDwtIDUgICAgICAjIHNsb3BlCgpzZXQuc2VlZCgxMjMpCihkZiA8LSB0aWJibGUoCiAgeCA9IHJ1bmlmKG4sIG1pbiA9IC0xLCBtYXggPSAxKSwKICB5ID0gYiArIGEqeCArIHJub3JtKG4pCikpCmBgYAoKIyMgU2ltcGxlIHJlZ3Jlc3Npb24gd2l0aCBPTFMKCkknbSBhc3N1bWluZyB3ZSdyZSBhbGwgZmFtaWxpYXIgd2l0aCB0aGUgT0xTIG1ldGhvZG9sb2d5IGZvciBsaW5lYXIgcmVncmVzc2lvbgptb2RlbGluZyB3aGVyZSBvdXIgb2JqZWN0aXZlIGZ1bmN0aW9uIChha2EgX19fbG9zc19fXykgaXM6CgokJGxvc3MgPSBNU0UgPSBcZnJhY3sxfXtufVxzdW1ebl97aS0xfShZX2kgLSBcaGF0IFlfaSleMiQkCgpXaXRoIE9MUywgd2UgY2FuIGZpbmQgdGhlIG1pbmltdW0gTVNFIHdpdGggYSBjbG9zZWQgZm9ybSBzb2x1dGlvbiAoZG9uJ3Qgd29ycnksCnRoaXMgZXF1YXRpb24gaXMgbm90IGltcG9ydGFudCDwn5isKToKCiQkIFxoYXQgXGJldGEgPSAoWF5UWCleey0xfVheVHkkJAoKd2hpY2ggd2lsbCBwcm92aWRlIHVzIHdpdGggb3VyIGNvZWZmaWNpZW50cyAoJGIkKSBpbiB0aGUgZXF1YXRpb246CgokJFxoYXQgeSA9IGJfMCArIGJfMXgkJAoKSWYgd2UgYXBwbHkgT0xTIHRvIG91ciBkYXRhLCB3ZSBnZXQ6CgoqIGVzdGltYXRlZCBpbnRlcmNlcHQgPSAzMC4wMQoqIGVzdGltYXRlZCBzbG9wZSA9IDQuOTcKKiBsb3NzIHNjb3JlIChNU0UpID0gMS4wMDIKCmBgYHtyfQpsbV9tb2RlbCA8LSBzdW1tYXJ5KGxtKHkgfiB4LCBkYXRhID0gZGYpKQpsbV9tb2RlbApgYGAKCldlIGNhbiBpbGx1c3RyYXRlIHRoaXMgbW9kZWwgZml0IHRvIG91ciBkYXRhOgoKYGBge3J9Cm1zZSA8LSBsbV9tb2RlbFtbInNpZ21hIl1dCgpnZ3Bsb3QoZGYsIGFlcyh4LCB5KSkgKwogIGdlb21fcG9pbnQoYWxwaGEgPSAwLjUpICsKICBnZW9tX3Ntb290aChtZXRob2QgPSAibG0iLCBzZSA9IEZBTFNFKSArCiAgZ2d0aXRsZShnbHVlKCJNU0UgPSB7bXNlfSIpKQpgYGAKCiMjIFNpbXBsZSByZWdyZXNzaW9uIHdpdGggYSBwZXJjZXB0cm9uCgpOb3csIGxldCdzIGlsbHVzdHJhdGUgcGVyZm9ybWluZyBhIHNpbWlsYXIgcHJvY2VzcyBidXQgd2l0aCBhIGJhc2ljIGJ1aWxkaW5nCmJsb2NrIG9mIG5ldXJhbCBuZXR3b3JrcyAtLSB0aGUgcGVyY2VwdHJvbi4KClRvIG1vZGVsIHdpdGgga2VyYXMgd2UgbmVlZCBvdXIgZGF0YSB0byBiZSBpbiBfX190ZW5zb3JzX19fLiBXZSdsbCBkaXNjdXNzCnRlbnNvcnMgbW9yZSBsYXRlciBidXQgZm9yIG5vdyBqdXN0IHJlYWxpemUgdGhhdDoKCiogMUQgdGVuc29yID0gdmVjdG9yCiogMkQgdGVuc29yID0gbWF0cml4CgpgYGB7cn0KeCA8LSBhcy5tYXRyaXgoZGYkeCkKeSA8LSBkZiR5CmBgYAoKVHJhaW5pbmcgYSBuZXVyYWwgbmV0d29yayBtb2RlbCBjb25zaXN0cyBvZiAzIHN0ZXBzOgoKMS4gRGVmaW5lIG1vZGVsIGFyY2hpdGVjdHVyZQoyLiBEZWZpbmUgaG93IG91ciBtb2RlbCBpcyBnb2luZyB0byBsZWFybgozLiBUcmFpbiBvdXIgbW9kZWwKCiMjIyAxLiBEZWZpbmUgbW9kZWwgYXJjaGl0ZWN0dXJlCgpEZWZpbmluZyBhbiBhcmNoaXRlY3R1cmUgaW5jbHVkZXMgZGVmaW5pbmcgdGhlIHR5cGUgb2YgbW9kZWwgYW5kIHRoZSBhcnJhbmdlbWVudApvZiBsYXllcnM6CgpgYGB7fQptb2RlbCA8LSBrZXJhc19tb2RlbF9zZXF1ZW50aWFsKCkgJT4lCiAgbGF5ZXJfZGVuc2UodW5pdHMgPSAxLCBpbnB1dF9zaGFwZSA9IG5jb2woeCkpCmBgYAoKKiBEZWZpbmUgc3RydWN0dXJlIG9mIG1vZGVsCiAgICAtIFNlcXVlbnRpYWw6IGxpbmVhciBsYXllcnMgKGkuZS4gTUxQLCBDTk4sIFJOTiwgTFNUTSkKICAgIC0gRnVuY3Rpb25hbDogYWRkcyBtb3JlIGZsZXhpYmlsaXR5IChpLmUuIGNvbWJpbmluZyBDTk4gJiBMU1RNIHRvIHByZWRpY3QKICAgICAgcHJvYmFiaWxpdHkgb2YgY2FuY2VyKQoqIERlZmluZSBhcnJhbmdlbWVudCBhbmQgc2hhcGUgb2YgbGF5ZXJzCiAgICAtIGBsYXllcl9kZW5zZWA6IGEgc2luZ2xlIGxheWVyIG9mIG5vZGVzCiAgICAtIGB1bml0cyA9IDFgOiBhIHNpbmdsZSBwZXJjZXB0cm9uIHVuaXQgaW4gb3VyIGxheWVyCiAgICAtIGBpbnB1dF9zaGFwZWA6IHdlIG5lZWQgdG8gdGVsbCBvdXIgZmlyc3QgbGF5ZXIgaG93IG1hbnkgaW5wdXRzIHRvIGV4cGVjdAoKIVtdKGltYWdlcy9wZXJjZXB0cm9uLnBuZykKCkFsbCB0aGlzIGlzIGNhbGN1bGF0aW5nIGlzIChpZiB3ZSBtYXRjaCB0aGUgYWJvdmUgaW1hZ2UpOgoKJCR5ID0gYmlhcyArIHdfMXhfMSQkCgojIyMgMi4gRGVmaW5lIGhvdyBvdXIgbW9kZWwgaXMgZ29pbmcgdG8gbGVhcm4KClRoZSB3aG9sZSBnb2FsIG9mIHRyYWluaW5nIGEgbmV1cmFsIG5ldHdvcmsgaXMgdG8gZmluZCB0aGUgb3B0aW1hbCBzZXQgb2YKcGFyYW1ldGVyIHdlaWdodHMgKGFrYSBjb2VmZmljaWVudHMgaW4gT0xTLXNwZWFrKS4gCgo+IF9fX091ciBnb2FsIGlzIHRvIGZpbmQgd2VpZ2h0cyB0aGF0IG1pbmltaXplIHRoZSBsb3NzIHNjb3JlX19fCgpXZSBkZWZpbmUgaG93IG91ciBtb2RlbCBpcyBnb2luZyB0byBsZWFybiB3aXRoIGBjb21waWxlKClgOgoKYGBge30KbW9kZWwgJT4lIGNvbXBpbGUoCiAgb3B0aW1pemVyID0gInNnZCIsCiAgbG9zcyA9ICJtc2UiCikKYGBgCgotIF9fT3B0aW1pemVyX186IG5ldXJhbCBuZXR3b3JrcyBsZWFybiB2aWEgX19fYmFja3Byb3BhZ2F0aW9uX19fLiBUaGVyZSBhcmUKICB2YXJpb3VzIF9fX2JhY2twcm9wYWdhdGlvbl9fXyBhbGdvcml0aG1zIGJ1dCB3ZSdsbCBzdGFydCB3aXRoIHRoZSBtb3N0IGJhc2ljLi4uCiAgc3RvY2hhc3RpYyBncmFkaWVudCBkZXNjZW50IChTR0QpLgogIAotIF9fbG9zc19fOiBob3cgZG8gd2Ugd2FudCB0byBtZWFzdXJlIG91ciBtb2RlbCdzIGVycm9yLiBrZXJhcyBjb21lcyB3aXRoIG1hbnkKICBidWlsdC1pbiBsb3NzIGZ1bmN0aW9ucyBhbmQgd2UgY2FuIGV2ZW4gY3JlYXRlIGN1c3RvbSBsb3NzIGZ1bmN0aW9ucy4gSGVyZSwKICB3ZSdsbCB1c2UgTVNFLgoKIVtdKGltYWdlcy9zZ2QucG5nKQoKCiMjIyAzLiBUcmFpbiBvdXIgbW9kZWwKClRoZSBsYXN0IHRoaW5nIHdlIG5lZWQgaXMgaG93IHRvIHRyYWluIG91ciBtb2RlbCwgd2hpY2ggd2UgZG8gd2l0aCBgZml0KClgOgoKYGBge30KaGlzdG9yeSA8LSBtb2RlbCAlPiUgZml0KHgsIHksIGJhdGNoX3NpemUgPSAzMiwgZXBvY2hzID0gMTApCmBgYAoKKiBgeGA6IGZlYXR1cmUgdGVuc29yCiogYHlgOiB0YXJnZXQgdGVuc29yCiogYGJhdGNoX3NpemVgOiBwaWNrIG9ic2VydmF0aW9ucyBmcm9tIG91ciB0cmFpbmluZyBkYXRhLCBwZXJmb3JtIGZvcndhcmQgcGFzcywKICAgY29tcHV0ZSBsb3NzIHNjb3JlLCBjb21wdXRlIGdyYWRpZW50LCBwZXJmb3JtIGJhY2t3YXJkIHBhc3MsIHVwZGF0ZSBvdXIKICAgd2VpZ2h0IChkZWZhdWx0ID0gMzIpLgoqIGBlcG9jaGA6IDEgZXBvY2ggPSBvbmUgZm9yd2FyZCBwYXNzIGFuZCBvbmUgYmFja3dhcmQgcGFzcyBvZiBhbGwgdGhlIHRyYWluaW5nCiAgIGV4YW1wbGVzLiBXZSdyZSByZXBlYXRpbmcgdGhhdCAxMCB0aW1lcyAoZGVmYXVsdCA9IDEwKS4KCgojIyMgUHV0dGluZyBpdCBhbGwgdG9nZXRoZXIKCkxldCdzIHB1dCBhbGwgdGhyZWUgc3RlcHMgdG9nZXRoZXIgYW5kIHRyYWluIG91ciBtb2RlbC4gSGVyZSdzIGEgdmlzdWFsCmRlcGljdGlvbjoKCiFbXShpbWFnZXMvcHV0X3RvZ2V0aGVyLnBuZykKCkFuZCBoZXJlJ3MgdGhlIGNvZGU6CgpgYGB7cn0KIyAxLiBEZWZpbmUgbW9kZWwgYXJjaGl0ZWN0dXJlCm1vZGVsIDwtIGtlcmFzX21vZGVsX3NlcXVlbnRpYWwoKSAlPiUKICBsYXllcl9kZW5zZSh1bml0cyA9IDEsIGlucHV0X3NoYXBlID0gbmNvbCh4KSkKCiMgMi4gRGVmaW5lIGhvdyBvdXIgbW9kZWwgaXMgZ29pbmcgdG8gbGVhcm4KbW9kZWwgJT4lIGNvbXBpbGUoCiAgb3B0aW1pemVyID0gInNnZCIsCiAgbG9zcyA9ICJtc2UiCikKCiMgMy4gVHJhaW4gb3VyIG1vZGVsCmhpc3RvcnkgPC0gbW9kZWwgJT4lIGZpdCh4LCB5LCBlcG9jaHMgPSAxMCkKYGBgCgpXaGVyZWFzIGluIE9MUyB3ZSBjYWxsIHRoZSBpbnRlcmNlcHQgYW5kIHNsb3BlIHBhcmFtZXRlcnMgImNvZWZmaWNpZW50cyIsIGluCm5ldXJhbCBuZXR3b3JrcyB3ZSBjYWxsIHRoZW0gX19fd2VpZ2h0c19fXy4gV2UgY2FuIHNlZSB0aGF0IGFmdGVyIDEwIGVwb2NocyBvdXIKd2VpZ2h0cyBhcmUgZ2V0dGluZyBjbG9zZSB0byB0aGUgdW5kZXJseWluZyAidHJ1dGgiIHZhbHVlcyAoc2xvcGUgPSA1LAppbnRlcmNlcHQgPSAzMCkgYnV0IGFsc28gbm90aWNlIHRoYXQgb3VyIE1TRSBsb3NzIHNjb3JlIGlzIHN0aWxsIGRlY3JlYXNpbmcKYWZ0ZXIgMTAgZXBvY2hzLgoKYGBge3J9CmdldF93ZWlnaHRzKG1vZGVsKQpgYGAKCiMjIyBNb2RlbHMgYXJlIG9iamVjdCBvcmllbnRlZAoKTm90ZSB0aGF0IG1vZGVsaW5nIHdpdGgga2VyYXMvdGVuc29yZmxvdyBpbiBSIG1heSBmZWVsIGEgYml0IGRpZmZlcmVudCB0aGFuCm90aGVyIG1vZGVsaW5nIHBhY2thZ2VzIHlvdSd2ZSB1c2VkIGluIFIuIFNpbmNlIGtlcmFzL3RlbnNvcmZsb3cgYXJlIHJldGljdWxhdGVkCmZyb20gUHl0aG9uLCB0aGUgbW9kZWwgaXMgYW4gX19fb2JqZWN0IG9yaWVudGVkX19fIG9iamVjdCB3aXRoIFB5dGhvbiBhdHRyaWJ1dGVzLgoKMS4gT2JqZWN0IG9yaWVudGVkIC0gb3VyIG1vZGVsIG9iamVjdCBjaGFuZ2VzIHdpdGhvdXQgYXNzaWdubWVudDoKICAgLSBgbW9kZWwgJT4lIGNvbXBpbGUoKWAgY2hhbmdlZCBvdXIgbW9kZWwgb2JqZWN0IGJ5IGFkZGluZyB0aGUgb3B0aW1pemVyIGFuZAogICAgIGxvc3MgcGFyYW1ldGVyIGFyZ3VtZW50cwogICAtIGBtb2RlbCAlPiUgZml0KClgIHdpbGwgY29udGludWUgdG8gYnVpbGQgb250byBvdXIgZXhpc3RpbmcgbW9kZWwKICAgCmBgYHtyfQojIGxldCdzIGV4ZWN1dGUgb25lIG1vcmUgZXBvY2guIE5vdGUgaG93IG91ciBsb3NzIGRlY3JlYXNlcyBmcm9tIHRoZSBsYXN0IGVwb2NoCiMgYWJvdmUKbW9kZWwgJT4lIGZpdCh4LCB5LCBlcG9jaCA9IDEpCgojIG91ciBtb2RlbCB3ZWlnaHRzIGdldCB1cGRhdGVkIGZyb20gdGhpcyBsYXN0IGVwb2NoIGFuZCBjb250aW51ZSB0byBnZXQgY2xvc2VyCiMgdG8gdGhlIHVuZGVybHlpbmcgdHJ1ZSB2YWx1ZXMKZ2V0X3dlaWdodHMobW9kZWwpCmBgYAoKMi4gT3VyIG1vZGVsIGlzIGEgUHl0aG9uIG9iamVjdCAtIHRoZXJlIHdpbGwgYmUgdGhpbmdzIHlvdSBjYW4gbm90IGRpcmVjdGx5CiAgIGFjY2VzcyBiZWNhdXNlIHRoZXkgYXJlIFB5dGhvbiBvYmplY3RzLiBIb3dldmVyLCBmb3IgbW9zdCB0aGluZ3MgdGhhdCB5b3Ugd2FudAogICB0byBhY2Nlc3MgdGhlcmUgd2lsbCBiZSBhIGZ1bmN0aW9uIHRvIGV4cG9ydCB0aGVtOgogICAKYGBge3J9CiMgdGhlIG1vZGVsIHdlaWdodHMgYXJlIGhlbGQgaW4gUHl0aG9uIG51bXB5IGFycmF5cwptb2RlbCR3ZWlnaHRzCgojIHdlIHVzZSBoZWxwZXIgZnVuY3Rpb25zIHRvIGV4cG9ydCB0aGVzZSBraW5kcyBvZiBvYmplY3RzCmdldF93ZWlnaHRzKG1vZGVsKQpgYGAKCgojIyBZb3VyIFR1cm4gKDMgbWluKQoKMS4gRmlsbCBpbiB0aGUgYmxhbmtzIGJlbG93IGFuZCB0cmFpbiB0aGUgbW9kZWwgZm9yIDI1IGVwb2Nocy4KMi4gRXhwbG9yZSB0aGUgYGhpc3RvcnlgIG9iamVjdC4KMy4gV2hhdCBhcmUgdGhlIGZpbmFsIHdlaWdodHMgZm9yIHRoaXMgbW9kZWw/IEhvdyBkbyB0aGV5IGNvbXBhcmUgdG8gdGhlCiAgIHVuZGVybHlpbmcgaW50ZXJjZXB0ICgzMCkgYW5kIHNsb3BlICg1KT8KCmBgYHtyfQojfCBldmFsOiBmYWxzZQoKIyAxLiBEZWZpbmUgbW9kZWwgYXJjaGl0ZWN0dXJlCm1vZGVsIDwtIGtlcmFzX21vZGVsX3NlcXVlbnRpYWwoKSAlPiUKICBsYXllcl9kZW5zZSh1bml0cyA9IF9fXywgaW5wdXRfc2hhcGUgPSBfX19fKQoKIyAyLiBEZWZpbmUgaG93IG91ciBtb2RlbCBpcyBnb2luZyB0byBsZWFybgptb2RlbCAlPiUgY29tcGlsZSgKICBvcHRpbWl6ZXIgPSAic2dkIiwKICBsb3NzID0gX19fXwopCgojIDMuIFRyYWluIG91ciBtb2RlbApoaXN0b3J5IDwtIG1vZGVsICU+JSBmaXQoeCwgeSwgZXBvY2hzID0gX19fXykKYGBgCgojIyBHcmFkaWVudCBkZXNjZW50CgpXZSBjYW4gc2VlIHRoZSBwcm9ncmVzc2lvbiBvZiB0aGUgZ3JhZGllbnQgZGVzY2VudCBwcm9jZXNzIGJ5IGV4YW1pbmluZyB0aGUKd2VpZ2h0cyBvdXIgbW9kZWwgcHJvZHVjZXMgYWZ0ZXIgZWFjaCBlcG9jaC4gVGhpcyBpcyBub3Qgc29tZXRoaW5nIHlvdSB3aWxsCmRvIG9mdGVuIGJ1dCwgcmF0aGVyLCBoZWxwcyBtYWtlIHRoZSBncmFkaWVudCBkZXNjZW50IHByb2Nlc3MgbW9yZSBjb25jcmV0ZS4KCmBgYHtyfQojIGRhdGEgZnJhbWUgdG8gZHVtcCBvdXIgcmVzdWx0cwptb2RlbF9lc3QgPC0gZXhwYW5kLmdyaWQoCiAgZXBvY2ggPSAxOjI1LAogIGFfc2dkID0gTkEsCiAgYl9zZ2QgPSBOQQopCgojIDEuIERlZmluZSBtb2RlbCBhcmNoaXRlY3R1cmUKbW9kZWwgPC0ga2VyYXNfbW9kZWxfc2VxdWVudGlhbCgpICU+JQogIGxheWVyX2RlbnNlKHVuaXRzID0gMSwgaW5wdXRfc2hhcGUgPSBuY29sKHgpKQoKIyAyLiBEZWZpbmUgaG93IG91ciBtb2RlbCBpcyBnb2luZyB0byBsZWFybgptb2RlbCAlPiUgY29tcGlsZSgKICBvcHRpbWl6ZXIgPSAic2dkIiwKICBsb3NzID0gIm1zZSIKKQoKIyAzLiBUcmFpbiBvdXIgbW9kZWwgZm9yIDI1IGVwb2NocyBhbmQgcmVjb3JkIHRoZSB3ZWlnaHRzIGFmdGVyIGVhY2ggZXBvY2gKZm9yIChyb3cgaW4gc2VxX2xlbihucm93KG1vZGVsX2VzdCkpKSB7CiAgCiAgaGlzdG9yeSA8LSBtb2RlbCAlPiUgZml0KHgsIHksIGVwb2NoID0gMSwgdmVyYm9zZSA9IEZBTFNFKQogIAogIGN1cnJlbnRfd3RzIDwtIGdldF93ZWlnaHRzKG1vZGVsKQoKICBtb2RlbF9lc3Rbcm93LCAiYV9zZ2QiXSA8LSBjdXJyZW50X3d0c1tbMV1dCiAgbW9kZWxfZXN0W3JvdywgImJfc2dkIl0gPC0gY3VycmVudF93dHNbWzJdXQp9CmBgYAoKVGhlIGZvbGxvd2luZyB0YWJsZSBzaG93cyB0aGUgZXN0aW1hdGUgc2xvcGUgKGBhX3NnZGApIGFuZCBpbnRlcmNlcHQgKGBiX3NnZGApCnByb2R1Y2VkIGJ5IFNHRCBhZnRlciBlYWNoIGVwb2NoLgoKYGBge3J9Cm1vZGVsX2VzdApgYGAKCmBgYHtyfQpoaXN0b3J5CmBgYAoKV2UgY2FuIHZpc3VhbGl6ZSBvdXIgbGluZWFyIG1vZGVsIGFmdGVyIGVhY2ggZXBvY2ggd2l0aCB0aGUgZm9sbG93aW5nLiBOb3RlIGhvdwplYWNoIGVwb2NoIHJlc3VsdHMgaW4gYSBsaW5lYXIgcHJlZGljdGlvbiAoZG90dGVkKSB0aGF0IGdldHMgY2xvc2VyIHRvIHRoZSB0cnV0aAooYmx1ZSBsaW5lKS4gQWZ0ZXIgMjUgZXBvY2hzIG91ciBtb2RlbCBiYXNpY2FsbHkgY29udmVyZ2VzIHRvIHRoZSBzYW1lIHJlc3VsdHMKKHllbGxvdyBkb3R0ZWQgbGluZSkuCgpXZSBjYW4gc2VlIHRoaXMgd2l0aCBvdXIgbG9zcyAoTVNFKSB0aGF0IG5lYXJseSBlcXVhdGVzIHRoZSBPTFMgTVNFICgxLjAwMTg3KS4KCmBgYHtyfQplcG9jaF9wcmVkIDwtIG1lcmdlKGRmLCBtb2RlbF9lc3QsIGFsbCA9IFRSVUUpICU+JQogIG11dGF0ZShwcmVkID0gYl9zZ2QgKyBhX3NnZCp4KQoKbGFzdF9lcG9jaCA8LSBmaWx0ZXIoZXBvY2hfcHJlZCwgZXBvY2ggPT0gbWF4KGVwb2NoKSkKCmdncGxvdChkYXRhID0gZGYsIGFlcyh4LCB5KSkgKwogIGdlb21fcG9pbnQoYWxwaGEgPSAwLjEpICsKICBnZW9tX3Ntb290aChtZXRob2QgPSAibG0iLCBzZSA9IEZBTFNFLCBzaXplID0gMS41KSArCiAgZ2VvbV9saW5lKGRhdGEgPSBlcG9jaF9wcmVkLCBhZXMoeCwgcHJlZCwgZ3JvdXAgPSBlcG9jaCksIGx0eSA9ICJkb3R0ZWQiKSArCiAgZ2VvbV9saW5lKGRhdGEgPSBsYXN0X2Vwb2NoLCBhZXMoeCwgcHJlZCwgZ3JvdXAgPSBlcG9jaCksIAogICAgICAgICAgICBsdHkgPSAiZG90dGVkIiwgY29sb3IgPSAieWVsbG93Iiwgc2l6ZSA9IDIpICsKICBnZ3RpdGxlKGdsdWUoIk1TRSA9IHtoaXN0b3J5JG1ldHJpY3MkbG9zc30iKSkKYGBgCgojIyBLZXkgVGFrZWF3YXlzCgoqIEEgYmFzaWMgc2luZ2xlIHBlcmNlcHRyb24gY29tcHV0ZXMgdGhlIHNhbWUgdHJhbnNmb3JtYXRpb24gYXMgT0xTOgoKJCRcaGF0IHkgPSBiaWFzICsgd2VpZ2h0XzEgXHRpbWVzIHhfMSArIHdlaWdodF8yIFx0aW1lcyB4XzIgKyBcZG90cyArIHdlaWdodF9uIFx0aW1lcyB4X24kJAoKKiBOZXVyYWwgbmV0d29ya3MgbGVhcm4gdmlhIGdyYWRpZW50IGRlc2NlbnQgLSBhbiBpdGVyYXRpdmUgYXBwcm9hY2ggb2YgcHJlZGljdGluZwogIHdpdGggYSBmb3J3YXJkIHBhc3MsIG1lYXN1cmluZyB0aGUgZ3JhZGllbnQgb2YgdGhlIGVycm9yLCBhbmQgcGVyZm9ybWluZyBhCiAgYmFja3dhcmQgcGFzcyB0byB1cGRhdGUgdGhlIHdlaWdodHMgYmFzZWQgb24gdGhlIGdyYWRpZW50LgoKCiMgQmluYXJ5IENsYXNzaWZpY2F0aW9uCgpMZXQncyBkbyB0aGUgc2FtZSBwcm9jZXNzIGJ1dCBub3cgd2UnbGwgZG8gc28gd2l0aCBhIGJpbmFyeSBjbGFzc2lmaWNhdGlvbgpwcm9ibGVtIChpLmUuIHByZWRpY3RpbmcgeWVzIHZzLiBubyByZXNwb25zZSkuCgpgYGB7cn0Kc2V0LnNlZWQoMTIzKQpnZW5lcmF0ZWQgPC0gbWxiZW5jaDo6bWxiZW5jaC5zaW1wbGV4KG4gPSAxMDAwLCBkID0gMSwgc2QgPSAuMykKeCA8LSBnZW5lcmF0ZWQkeAp5IDwtIGlmZWxzZShnZW5lcmF0ZWQkY2xhc3NlcyA9PSAxLCAwLCAxKQoKKGRmIDwtIHRpYmJsZSh4ID0gYXMudmVjdG9yKHgpLCB5ID0geSkpCmBgYAoKT3VyIGdlbmVyYXRlZCBkYXRhIGhhcyBzb21lIG92ZXJsYXAgc28gdGhlcmUgaXMgbm8gbGluZWFyIHNlcGVyYXRpb24gd2l0aG91dApoYXZpbmcgc29tZSBlcnJvci4gTm90ZSB0aGFuIHdoZW4gZGlzY3Vzc2luZyBiaW5hcnkgY2xhc3NpZmljYXRpb24gcHJvYmxlbXMsIHdlCndpbGwgbWFpbmx5IHVzZSB0aGUgY3Jvc3NlbnRyb3B5IChha2EgbG9nIGxvc3MpIGxvc3MgZnVuY3Rpb24uCgpgYGB7cn0KZ2xtX21vZGVsIDwtIGdsbSh5IH4geCwgZmFtaWx5ID0gYmlub21pYWwobGluayA9ICJsb2dpdCIpLCBkYXRhID0gZGYpCmNyb3NzZW50cm9weSA8LSBNTG1ldHJpY3M6OkxvZ0xvc3MoZ2xtX21vZGVsJGZpdHRlZC52YWx1ZXMsIGRmJHkpCgpnZ3Bsb3QoZGYsIGFlcyh4LCB5KSkgKwogIGdlb21fcG9pbnQoYWVzKGNvbG9yID0gYXMuZmFjdG9yKHkpKSwgc2l6ZSA9IDIsIHNob3cubGVnZW5kID0gRkFMU0UpICsKICBnZW9tX3Ntb290aChtZXRob2QgPSAiZ2xtIiwgbWV0aG9kLmFyZ3MgPSBsaXN0KGZhbWlseSA9ICJiaW5vbWlhbCIpLCBzZSA9IEZBTFNFKSArCiAgZ2d0aXRsZShnbHVlKCJjcm9zc2VudHJvcHkgPSB7Y3Jvc3NlbnRyb3B5fSIpKSArCiAgeWxhYigicHJvYmFiaWxpdHkgeSA9IDEiKQpgYGAKCiMjIFNpZ21vaWQgQWN0aXZhdGlvbiBGdW5jdGlvbgoKV2hlbiBwcmVkaWN0aW5nIGEgYmluYXJ5IHJlc3BvbnNlLCB3ZSB0eXBpY2FsbHkgd2FudCB0byBwcmVkaWN0IGEgcmVhbCB2YWx1ZQpiZXR3ZWVuIDAtMSByZXByZXNlbnRpbmcgdGhlIHByb2JhYmlsaXR5IG9mIHRoZSBwb3NpdGl2ZSBiaW5hcnkgY2xhc3MuClVuZm9ydHVuYXRlbHkgb3VyIHJlZ3VsYXIgcGVyY2VwdHJvbiBjcmVhdGVzIGEgbGluZWFyIHRyYW5zZm9ybWF0aW9uLiBIb3dldmVyLAp3ZSBjYW4gYXBwbHkgYW4gX19fYWN0aXZhdGlvbiBmdW5jdGlvbl9fXyB0byB0cmFuc2Zvcm0gdGhpcyBsaW5lYXIgdHJhbnNmb3JtYXRpb24KdG8gYSBub24tbGluZWFyIHRyYW5zZm9ybWF0aW9uLgoKV2hlbiBwcmVkaWN0aW5nIGEgYmluYXJ5IHJlc3BvbnNlLCB3ZSB1c2UgYSBfX19zaWdtb2lkX19fIGFjdGl2YXRpb24gdG8gY29udmVydApvdXIgbGluZWFyIHRyYW5zZm9ybWF0aW9uIHRvIGEgMC0xIHByb2JhYmlsaXR5IG9mIHRoZSBwb3NpdGl2ZSBjbGFzcy4KCiQkc2lnbW9pZCh5KSA9IFxmcmFjezF9ezErZV57LXl9fSQkCgohW10oaW1hZ2VzL3NpZ21vaWQuanBnKQoKV2hlbiBwcmVkaWN0aW5nIGEgYmluYXJ5IHJlc3BvbnNlLCB3ZSBuZWVkIHRvIG1ha2UgdGhlIGZvbGxvd2luZyBjaGFuZ2VzIHRvIG91cgpjb2RlOgoKKiBBZGQgYGFjdGl2YXRpb24gPSAic2lnbW9pZCJgIHRvIHRoZSBsYXllciB0aGF0IGlzIHByZWRpY3RpbmcgdGhlIG91dHB1dC4KKiBOb3RlIHRoYXQgc2luY2Ugd2UgYXJlIHByZWRpY3RpbmcgdGhlIHByb2JhYmlsaXR5IGZyb20gMC0xIGZvciBvdXIgcmVzcG9uc2UsCiAgd2Uga2VlcCBgdW5pdHMgPSAxYC4KKiBsb3NzIC0gd2UgY2hhbmdlIGBsb3NzID0gImJpbmFyeV9jcm9zc2VudHJvcHkiYCB0byB1c2UgdGhlIGNyb3NzZW50cm9weSAvIGxvZwogIGxvc3Mgb2JqZWN0aXZlIGZ1bmN0aW9uLgoKYGBge3J9Cm1vZGVsIDwtIGtlcmFzX21vZGVsX3NlcXVlbnRpYWwoKSAlPiUKICBsYXllcl9kZW5zZSh1bml0cyA9IDEsIGlucHV0X3NoYXBlID0gbmNvbCh4KSwgYWN0aXZhdGlvbiA9ICJzaWdtb2lkIikKCm1vZGVsICU+JSBjb21waWxlKAogIG9wdGltaXplciA9ICJzZ2QiLAogIGxvc3MgPSAiYmluYXJ5X2Nyb3NzZW50cm9weSIKKQoKKGhpc3RvcnkgPC0gbW9kZWwgJT4lIGZpdCh4LCB5LCBlcG9jaHMgPSA1MCwgdmVyYm9zZSA9IEZBTFNFKSkKYGBgCgpXZSBzZWUgdGhhdCBvdXIgbG9zcyBpcyBxdWl0ZSBhIGJpdCBvZmYgZnJvbSBvdXIgbG9naXN0aWMgcmVncmVzc2lvbiBtb2RlbC4KCmBgYHtyfQpkZiAlPiUKICBtdXRhdGUocHJlZCA9IHByZWRpY3QobW9kZWwsIHgpICU+JSBhcy52ZWN0b3IoKSkgJT4lCiAgZ2dwbG90KGFlcyh4LCB5KSkgKwogIGdlb21fcG9pbnQoYWVzKGNvbG9yID0gYXMuZmFjdG9yKHkpKSwgc2l6ZSA9IDIsIHNob3cubGVnZW5kID0gRkFMU0UpICsKICBnZW9tX3Ntb290aChtZXRob2QgPSAiZ2xtIiwgbWV0aG9kLmFyZ3MgPSBsaXN0KGZhbWlseSA9ICJiaW5vbWlhbCIpLCBzZSA9IEZBTFNFKSArCiAgZ2VvbV9saW5lKGFlcyh5ID0gcHJlZCksIGx0eSA9ICJkYXNoZWQiKSArCiAgZ2d0aXRsZShnbHVlKCJjcm9zc2VudHJvcHkgPSB7bWluKGhpc3RvcnkkbWV0cmljcyRsb3NzKX0iKSkgKwogIHlsYWIoInByb2JhYmlsaXR5IG9mIHkgPSAxIikKYGBgCgpIb3dldmVyLCBpZiB3ZSBsb29rIGF0IG91ciBsb3NzIHNjb3Jlcywgd2Ugc2VlIHRoYXQgdGhleSBhcmUgc3RpbGwgaW1wcm92aW5nLAppdHMganVzdCB0YWtpbmcgYSBsb25nIHRpbWUuIFBsdXMsIGl0IGxvb2tzIGxpa2UgdGhlcmUgaXMgbW9yZSBpbXByb3ZlbWVudCB0aGF0CmNhbiBiZSBtYWRlIHRvIG91ciBsb3NzLgoKYGBge3J9CnBsb3QoaGlzdG9yeSkKYGBgCgojIyBMZWFybmluZyByYXRlIGFuZCBtb21lbnR1bQoKQW4gaW1wb3J0YW50IHBhcmFtZXRlciBpbiBncmFkaWVudCBkZXNjZW50IGlzIHRoZSBzaXplIG9mIHRoZSBzdGVwcyB3aGljaCBpcwpjb250cm9sbGVkIGJ5IHRoZSBfX19sZWFybmluZyByYXRlX19fLiBJZiB0aGUgbGVhcm5pbmcgcmF0ZSBpcy4uLgoKKiB0b28gc21hbGw6IHRoZSBhbGdvcml0aG0gd2lsbCB0YWtlIG1hbnkgaXRlcmF0aW9ucyAoc3RlcHMpIHRvIGZpbmQgdGhlIG1pbmltdW0KKiB0b28gbGFyZ2U6IHlvdSBtaWdodCBqdW1wIGFjcm9zcyB0aGUgbWluaW11bSBhbmQgZW5kIHVwIGZ1cnRoZXIgYXdheSB0aGFuIHdoZW4KICB5b3Ugc3RhcnRlZAogIAohW10oaW1hZ2VzL2xyLnBuZykKClRoZSBkZWZhdWx0IGxlYXJuaW5nIHJhdGUgZm9yIFNHRCBpcyAwLjAxLiBVbmZvcnR1bmF0ZWx5IHdpdGggdGhpcyByYXRlLCBpdCB3aWxsCnRha2Ugb3ZlciAxLDAwMCBlcG9jaHMgdG8gcmVhY2ggYSBsb3NzIHNjb3JlIGNvbXBhcmFibGUgdG8gbG9naXN0aWMgcmVncmVzc2lvbi4KSG93ZXZlciwgd2UgY2FuIGN1c3RvbWl6ZSBvdXIgb3B0aW1pemVyIHdpdGggYG9wdGltaXplcl9zZGcoKWA6CgpgYGB7cn0KbW9kZWwgPC0ga2VyYXNfbW9kZWxfc2VxdWVudGlhbCgpICU+JQogIGxheWVyX2RlbnNlKHVuaXRzID0gMSwgaW5wdXRfc2hhcGUgPSBuY29sKHgpLCBhY3RpdmF0aW9uID0gInNpZ21vaWQiKQoKbW9kZWwgJT4lIGNvbXBpbGUoCiAgb3B0aW1pemVyID0gb3B0aW1pemVyX3NnZChsciA9IDAuMSksCiAgbG9zcyA9ICJiaW5hcnlfY3Jvc3NlbnRyb3B5IgopCgooaGlzdG9yeSA8LSBtb2RlbCAlPiUgZml0KHgsIHksIGVwb2NocyA9IDUwLCB2ZXJib3NlID0gRkFMU0UpKQpgYGAKCkFub3RoZXIgY29tbW9uIGFwcHJvYWNoIHRvIGFkanVzdCBvdXIgbGVhcm5pbmcgcmF0ZSBpcyB0byBhZGQgX19fbW9tZW50dW1fX18uCkFkZGluZyBtb21lbnR1bSBhbGxvd3Mgb3VyIGxlYXJuaW5nIHJhdGUgdG8gYWRhcHQuIE1vbWVudHVtIHNpbXBseSBhZGRzIGEKZnJhY3Rpb24gb2YgdGhlIHByZXZpb3VzIHdlaWdodCB1cGRhdGUgdG8gdGhlIGN1cnJlbnQgb25lLgoKIVtdKGltYWdlcy9tb21lbnR1bS5naWYpCgpMZXQncyBhZGQgc29tZSBtb21lbnR1bSB0byBvdXIgbGVhcm5pbmcgcmF0ZS4gV2Ugc2VlIHRoYXQgb3VyIGxvc3MgaW1wcm92ZXMKZXZlbiBtb3JlIHdpdGhpbiB0aGUgc2FtZSBudW1iZXIgb2YgZXBvY2hzLgoKYGBge3J9Cm1vZGVsIDwtIGtlcmFzX21vZGVsX3NlcXVlbnRpYWwoKSAlPiUKICBsYXllcl9kZW5zZSh1bml0cyA9IDEsIGlucHV0X3NoYXBlID0gbmNvbCh4KSwgYWN0aXZhdGlvbiA9ICJzaWdtb2lkIikKCm1vZGVsICU+JSBjb21waWxlKAogIG9wdGltaXplciA9IG9wdGltaXplcl9zZ2QobHIgPSAwLjEsIG1vbWVudHVtID0gMC41KSwKICBsb3NzID0gImJpbmFyeV9jcm9zc2VudHJvcHkiCikKCihoaXN0b3J5IDwtIG1vZGVsICU+JSBmaXQoeCwgeSwgZXBvY2hzID0gNTAsIHZlcmJvc2UgPSBGQUxTRSkpCmBgYAoKIyMgWW91ciBUdXJuICgzIG1pbikKCjEuIFRyeSBkaWZmZXJlbnQgY29tYmluYXRpb25zIG9mIGxlYXJuaW5nIHJhdGUgYW5kIG1vbWVudHVtLiBBIGZldyBydWxlcyBvZiDwn5GNOgogICAqIFR5cGljYWxseSwgd2Ugc3RhcnQgYXNzZXNzaW5nIGxlYXJuaW5nIHJhdGVzIGluIGxvZyB2YWx1ZXMgcmFuZ2VzIG9mIFsxZS0xLCAxZS03XQogICAgIChpLmUuIDAuMSwgMC4wMSwgLi4uLCAwLjAwMDAwMDEpLgogICAqIE1vbWVudHVtIGlzIHR5cGljYWxseSA+IDAuNSBhbmQgb2Z0ZW4gaW4gdGhlIDAuOS0wLjk5IHJhbmdlLgoyLiBQbG90IHRoZSBsb3NzIGxlYXJuaW5nIGN1cnZlLgozLiBIb3cgZG9lcyB5b3VyIGZpbmFsIGxvc3MgY29tcGFyZSB0byBsb2dpc3RpYyByZWdyZXNzaW9uPwoKYGBge3J9Cm1vZGVsIDwtIGtlcmFzX21vZGVsX3NlcXVlbnRpYWwoKSAlPiUKICBsYXllcl9kZW5zZSh1bml0cyA9IDEsIGlucHV0X3NoYXBlID0gbmNvbCh4KSwgYWN0aXZhdGlvbiA9ICJzaWdtb2lkIikKCm1vZGVsICU+JSBjb21waWxlKAogIG9wdGltaXplciA9IG9wdGltaXplcl9zZ2QobHIgPSAwLjEsIG1vbWVudHVtID0gLjk1KSwKICBsb3NzID0gImJpbmFyeV9jcm9zc2VudHJvcHkiCikKCihoaXN0b3J5IDwtIG1vZGVsICU+JSBmaXQoeCwgeSwgZXBvY2hzID0gNTAsIHZlcmJvc2UgPSBGQUxTRSkpCmBgYAoKCiMjIEtleSB0YWtlYXdheXMKCiogQWN0aXZhdGlvbiBmdW5jdGlvbnM6CiAgIC0gV2UgdXNlIGFjdGl2YXRpb24gZnVuY3Rpb25zIHRvIHRyYW5zZm9ybSB0aGUgcGVyY2VwdHJvbidzIGxpbmVhciBlcXVhdGlvbgogICAgIHRvIGEgbm9uLWxpbmVhciBmb3JtLgogICAtIEZvciBiaW5hcnkgY2xhc3NpZmljYXRpb24gcHJvYmxlbXMgd2UgdXNlIHRoZSAic2lnbW9pZCIgYWN0aXZhdGlvbiB0bwogICAgIGNvbnZlcnQgb3VyIHByZWRpY3Rpb25zIHRvIGEgMC0xIHByb2JhYmlsaXR5LgoqIExlYXJuaW5nIHJhdGU6CiAgIC0gV2UgY2FuIGNvbnRyb2wgdGhlIHJhdGUgb2YgbGVhcm5pbmcgYnkgaW5jcmVhc2luZyAmIGRlY3JlYXNpbmcgdGhlIGxlYXJuaW5nCiAgICAgcmF0ZS4KICAgLSBXZSBjYW4gbWFrZSB0aGlzIGxlYXJuaW5nIHJhdGUgYWRhcHRpdmUgdG8gdGhlIGN1cnZhdHVyZSBvZiBvdXIgbG9zcwogICAgIGdyYWRpZW50IGJ5IGluY29ycG9yYXRpbmcgbW9tZW50dW0uCgoKIyBOb24tbGluZWFyIFBhdHRlcm5zCgpBcyBvdXIgZGF0YXNldHMgZ2V0IGxhcmdlciBvciBpbmNsdWRlIG5vbi1saW5lYXJpdGllcywgb3VyIG1vZGVsIG5lZWRzIHRvIGJlY29tZQptb3JlIHNvcGhpc3RpY2F0ZWQuIEZvciB0aGlzIGV4YW1wbGUsIHdlJ2xsIHN0aWNrIHdpdGggb25lIHByZWRpY3RvciB2YXJpYWJsZQpidXQgd2UnbGwgYWRkIGEgbm9uLWxpbmVhcml0eSBjb21wb25lbnQ6CgpgYGB7cn0Kc2V0LnNlZWQoMTIzKQpkZiA8LSB0aWJibGUoCiAgeCA9IHNlcShmcm9tID0gLTEsIHRvID0gMiAqIHBpLCBsZW5ndGggPSBuKSwKICBlID0gcm5vcm0obiwgc2QgPSAwLjIpLAogIHkgPSBzaW4oeCkgKyBlCikKCmdncGxvdChkZiwgYWVzKHgsIHkpKSArCiAgZ2VvbV9wb2ludChhbHBoYSA9IDAuNSkgKwogIGdlb21fc21vb3RoKHNlID0gRkFMU0UpCmBgYAoKQWdhaW4sIGxldCdzIGV4dHJhY3Qgb3VyIGZlYXR1cmUgYW5kIHRhcmdldCB0ZW5zb3JzOgoKYGBge3J9CnggPC0gYXMubWF0cml4KGRmJHgpCnkgPC0gZGYkeQpgYGAKCkFzIG91ciB1bmRlcmx5aW5nIG1vZGVsIGhhcyBtb3JlIGNvbXBsZXhpdHksIHdlIGFkZCBoaWRkZW4gbGF5ZXJzIHRvIGNhcHR1cmUKbm9uLWxpbmVhcml0aWVzIGFuZCBpbnRlcmFjdGlvbnMuIFdlIGNhbGwgdGhlc2UgbmV1cmFsIG5ldHdvcmsgbW9kZWxzIApfX19tdWx0aS1sYXllciBwZXJjZXB0cm9uc19fXyAoTUxQcyk7IGFsc28gcmVmZXJyZWQgdG8gYXMgX19fZGVuc2VseSBjb25uZWN0ZWQKZmVlZCBmb3J3YXJkX19fIG5ldHdvcmtzLgoKIVtdKGltYWdlcy9iYXNpY19tbHAucG5nKQoKV2UgY2FuIGFkZCBhIGhpZGRlbiBsYXllcnMgYnkgYWRkaW5nIGFkZGl0aW9uYWwgYGxheWVyX2RlbnNlKClgIGZ1bmN0aW9ucyB0byBvdXIKbW9kZWwgYXJjaGl0ZWN0dXJlLiBGb3IgZXhhbXBsZSwgdGhlIGZvbGxvd2luZyBjb2RlIHdvdWxkIGNyZWF0ZSBhbiBNTFAgd2l0aDoKCiogMyBoaWRkZW4gbGF5ZXJzOiAKICAgLSBlYWNoIGhpZGRlbiBsYXllciBoYXMgMTYgbm9kZXMKICAgLSBvbmx5IHRoZSBmaXJzdCBoaWRkZW4gbGF5ZXIgcmVxdWlyZXMgYGlucHV0X3NoYXBlYAogICAtIGVhY2ggaGlkZGVuIGxheWVyIHVzZXMgYSBSZUxVIGFjdGl2YXRpb24gZnVuY3Rpb24gKHdlJ2xsIGRpc2N1c3Mgc2hvcnRseSkKKiB0aGUgbGFzdCBgbGF5ZXJfZGVuc2UoKWAgaXMgYWx3YXlzIHRoZSBvdXRwdXQgbGF5ZXIKICAgLSBhY3RpdmF0aW9uIGZ1bmN0aW9uIGZvciBvdXRwdXQgbGF5ZXIgaXMgYWx3YXlzIGRlcGVuZGVudCBvbiB0aGUgcHJvYmxlbQogICAgICAtIHJlZ3Jlc3Npb246IE5VTEwKICAgICAgLSBiaW5hcnkgY2xhc3NpZmljYXRpb246IGBhY3RpdmF0aW9uID0gInNpZ25tb2lkImAKICAgICAgLSBtdWx0aS1jbGFzcyBjbGFzc2lmaWNhdGlvbjogYGFjdGl2YXRpb24gPSAic29mdG1heCJgCgpgYGB7fQptb2RlbCA8LSBrZXJhc19tb2RlbF9zZXF1ZW50aWFsKCkgJT4lCiAgbGF5ZXJfZGVuc2UodW5pdHMgPSAxNiwgaW5wdXRfc2hhcGUgPSBuY29sKHgpLCBhY3RpdmF0aW9uID0gInJlbHUiKSAlPiUKICBsYXllcl9kZW5zZSh1bml0cyA9IDE2LCBhY3RpdmF0aW9uID0gInJlbHUiKSAlPiUKICBsYXllcl9kZW5zZSh1bml0cyA9IDE2LCBhY3RpdmF0aW9uID0gInJlbHUiKSAlPiUKICBsYXllcl9kZW5zZSh1bml0cyA9IDEpCmBgYAoKIyMgV2h5IFJlTFUKClRoZSByZWN0aWZpZWQgbGluZWFyIGFjdGl2YXRpb24gZnVuY3Rpb24gaXMgdmVyeSBzaW1wbGU7IGlmIHRoZSBsaW5lYXIKdHJhbnNmb3JtYXRpb24gd2l0aGluIHRoZSBwZXJjZXB0cm9uIHJlc3VsdHMgaW4gYSBuZWdhdGl2ZSBudW1iZXIgdGhlbiB0aGUKb3V0cHV0IGlzIDAuIElmIGl0cyBwb3NpdGl2ZSB0aGVuIGl0cyB0aGF0IHZhbHVlLgoKJCRSZUxVID0gbWF4KDAsIHopJCQKCiFbXShpbWFnZXMvUmVMVS5wbmcpCgpCZW5lZml0cyAoc2VlIGh0dHA6Ly9wcm9jZWVkaW5ncy5tbHIucHJlc3MvdjE1L2dsb3JvdDExYS9nbG9yb3QxMWEucGRmKToKCi0gU2ltcGxlIGdlb21ldHJpYyB0cmFuc2Zvcm1hdGlvbnMgY2FuIHByb2R1Y2UgdmVyeSBjb21wbGV4IHBhdHRlcm5zLgotIENvbXB1dGF0aW9uYWwgc2ltcGxpY2l0eSAoZWFzeSB0byBjb21wdXRlIHRoZSBncmFkaWVudCkKLSBSZXByZXNlbnRhdGlvbmFsIHNwYXJjaXR5IChmb3JjaW5nIDBzIHJlc3VsdHMgaW4gc3BhcnNlIG91dHB1dHMpCi0gTGluZWFyaXR5IChyZWR1Y2VzIHZhbmlzaGluZyBncmFkaWVudCBkZXNjZW50IC0gZGlzY3Vzc2VkIGxhdGVyKQoKIVtdKGltYWdlcy9vcmlnYW1pLmdpZikKCkxldCdzIHNlZSB0aGlzIGluIGFjdGlvbjoKCmBgYHtyfQojIGxpYnJhcnkocmV0aWN1bGF0ZSkKIyBsaWJyYXJ5KGtlcmFzKQojIHVzZV9jb25kYWVudigici10ZW5zb3JmbG93IikKCm1vZGVsIDwtIGtlcmFzX21vZGVsX3NlcXVlbnRpYWwoKSAlPiUKICBsYXllcl9kZW5zZSh1bml0cyA9IDE2LCBpbnB1dF9zaGFwZSA9IG5jb2woeCksIGFjdGl2YXRpb24gPSAicmVsdSIpICU+JQogIGxheWVyX2RlbnNlKHVuaXRzID0gMSkKCm1vZGVsICU+JSBjb21waWxlKAogIG9wdGltaXplciA9IG9wdGltaXplcl9zZ2QobHIgPSAwLjAxLCBtb21lbnR1bSA9IC45KSwKICBsb3NzID0gIm1zZSIKKQoKKGhpc3RvcnkgPC0gbW9kZWwgJT4lIGZpdCh4LCB5LCBlcG9jaHMgPSA1MCwgdmVyYm9zZSA9IEZBTFNFKSkKYGBgCgpgYGB7cn0KZGYgJT4lCiAgbXV0YXRlKHByZWQgPSBwcmVkaWN0KG1vZGVsLCB4KSAlPiUgYXMudmVjdG9yKCkpICU+JQogIGdncGxvdChhZXMoeCwgeSkpICsKICBnZW9tX3BvaW50KGFscGhhID0gMC4wNSkgKwogIGdlb21fc21vb3RoKHNlID0gRkFMU0UpICsKICBnZW9tX2xpbmUoYWVzKHkgPSBwcmVkKSwgbHR5ID0gImRhc2hlZCIsIGNvbG9yID0gInJlZCIsIHNpemUgPSAxKQpgYGAKCiMjIE1vZGVsIGNhcGFjaXR5CgpfX19Nb2RlbCBjYXBhY2l0eV9fXyBkZXRlcm1pbmVzIHRoZSBleHRlbmQgdG8gd2hpY2ggb3VyIG1vZGVsIGNhbiBjYXB0dXJlCnVuZGVybHlpbmcgcmVsYXRpb25zaGlwcyBhbmQgcGF0dGVybnMuIFdlIGNvbnRyb2wgbW9kZWwgY2FwYWNpdHkgd2l0aDoKCjEuIF9fd2lkdGhfXzogbnVtYmVyIG9mIHVuaXRzIGluIGEgbGF5ZXIKICAgLSBSdWxlIG9mIPCfkY06IHR5cGljYWxseSB1c2UgcG93ZXJzIG9mIDIgKGkuZS4gMTYsIDMyLCA2NCwgMTI4LCAyNTYsIDUxMikKMi4gX19kZXB0aF9fOiBudW1iZXIgb2YgaGlkZGVuIGxheWVycwogICAtIFJ1bGUgb2Yg8J+RjTogd2Ugb2Z0ZW4gc2VlIGJldHRlciBwZXJmb3JtYW5jZSAoYWNjdXJhY3kgJiBjb21wdXRlIGVmZmljaWVuY3kpCiAgICAgYnkgaW5jcmVhc2luZyB0aGUgbnVtYmVyIG9mIGxheWVycyBtb3Jlc28gdGhhbiBub2Rlcy4gCiAgICAgCkxldCdzIGFkZCAyIGhpZGRlbiBsYXllcnMsIGVhY2ggd2l0aCAxNiB1bml0czoKCmBgYHtyfQptb2RlbCA8LSBrZXJhc19tb2RlbF9zZXF1ZW50aWFsKCkgJT4lCiAgbGF5ZXJfZGVuc2UodW5pdHMgPSAxNiwgaW5wdXRfc2hhcGUgPSBuY29sKHgpLCBhY3RpdmF0aW9uID0gInJlbHUiKSAlPiUKICBsYXllcl9kZW5zZSh1bml0cyA9IDE2LCBhY3RpdmF0aW9uID0gInJlbHUiKSAlPiUKICBsYXllcl9kZW5zZSh1bml0cyA9IDEpCgptb2RlbCAlPiUgY29tcGlsZSgKICBvcHRpbWl6ZXIgPSBvcHRpbWl6ZXJfc2dkKGxyID0gMC4wMSwgbW9tZW50dW0gPSAuOSksCiAgbG9zcyA9ICJtc2UiCikKCihoaXN0b3J5IDwtIG1vZGVsICU+JSBmaXQoeCwgeSwgZXBvY2hzID0gNTAsIHZlcmJvc2UgPSBGQUxTRSkpCmBgYAoKTG9va2luZyBhdCBob3cgb3VyIHByZWRpY3RlZCB2YWx1ZXMgZml0IHRoZSB0cnVlIHVuZGVybHlpbmcgbW9kZWw6CgpgYGB7cn0KZGYgJT4lCiAgbXV0YXRlKHByZWQgPSBwcmVkaWN0KG1vZGVsLCB4KSAlPiUgYXMudmVjdG9yKCkpICU+JQogIGdncGxvdChhZXMoeCwgeSkpICsKICBnZW9tX3BvaW50KGFscGhhID0gMC4wNSkgKwogIGdlb21fc21vb3RoKHNlID0gRkFMU0UpICsKICBnZW9tX2xpbmUoYWVzKHkgPSBwcmVkKSwgbHR5ID0gImRhc2hlZCIsIGNvbG9yID0gInJlZCIsIHNpemUgPSAxKQpgYGAKCiMjIFlvdXIgVHVybiAoMyBtaW4pCgoxLiBUcnkgdXNpbmcgb25seSBvbmUgaGlkZGVuIGxheWVyIGFuZCBpbmNyZWFzZSB0aGUgd2lkdGggdG8gMzIsIDY0LCAxMjgsIDI1NgoyLiBUcnkgYWRkaW5nIGEgc2Vjb25kIGxheWVyIGFuZCBpbmNyZWFzZSB0aGUgd2lkdGggb2YgZWFjaCBsYXllciBwcm9ncmVzc2l2ZWx5CiAgIC0gUnVsZSBvZiDwn5GNOiB3aGVuIHdlIGFkZCBtb3JlIGxheWVycyB3ZSB0eXBpY2FsbHkgaGF2ZSB0aGUgZm9sbG93aW5nIHBhdHRlcm5zOgogICAgICAtIHR1bm5lbCBzaGFwZWQ6IGVhY2ggaGlkZGVuIGxheWVyIGhhcyB0aGUgc2FtZSBudW1iZXIgb2YgdW5pdHMKICAgICAgLSBmdW5uZWwgc2hhcGVkOiBoaWRkZW4gbGF5ZXJzIHByb2dyZXNzaXZlbHkgZ2V0IHNtYWxsZXIKCmBgYHtyfQojfCBldmFsOiBmYWxzZQoKCm1vZGVsIDwtIGtlcmFzX21vZGVsX3NlcXVlbnRpYWwoKSAlPiUKICBsYXllcl9kZW5zZSh1bml0cyA9IF9fX18sIGlucHV0X3NoYXBlID0gX19fXywgYWN0aXZhdGlvbiA9IF9fX18pICU+JQogIGxheWVyX2RlbnNlKHVuaXRzID0gMSkKCm1vZGVsICU+JSBjb21waWxlKAogIG9wdGltaXplciA9IG9wdGltaXplcl9zZ2QobHIgPSAwLjAxLCBtb21lbnR1bSA9IC45KSwKICBsb3NzID0gIm1zZSIKKQoKKGhpc3RvcnkgPC0gbW9kZWwgJT4lIGZpdCh4LCB5LCBlcG9jaHMgPSA1MCwgdmVyYm9zZSA9IEZBTFNFKSkKYGBgCgpSdW4gdGhlIGZvbGxvd2luZyB0byBzZWUgaG93IHlvdXIgYWRqdXN0ZWQgbW9kZWwgZml0cyB0aGUgYWN0dWFsIGRhdGE6CgpgYGB7cn0KZGYgJT4lCiAgbXV0YXRlKHByZWQgPSBwcmVkaWN0KG1vZGVsLCB4KSAlPiUgYXMudmVjdG9yKCkpICU+JQogIGdncGxvdChhZXMoeCwgeSkpICsKICBnZW9tX3BvaW50KGFscGhhID0gMC4yNSkgKwogIGdlb21fc21vb3RoKHNlID0gRkFMU0UpICsKICBnZW9tX2xpbmUoYWVzKHkgPSBwcmVkKSwgbHR5ID0gImRhc2hlZCIsIGNvbG9yID0gInJlZCIsIHNpemUgPSAxKQpgYGAKCiMjIEtleSBUYWtlYXdheXMKCiogSGlkZGVuIGxheWVycyBhbG1vc3QgYWx3YXlzIHVzZSB0aGUgUmVMVSBhY3RpdmF0aW9uIGZ1bmN0aW9uICh0aGlzIHNob3VsZCBiZQogIHlvdXIgZGVmYXVsdCkuCiogQ29udHJvbCBtb2RlbCBjYXBhY2l0eSBieSB3aWR0aCBhbmQgZGVwdGggKGFkZGluZyBkZXB0aCB0eXBpY2FsbHkgb3V0cGVyZm9ybXMKICBzaW1wbHkgZm9jdXNpbmcgb24gd2lkdGgpLgoKIyBNdWx0aS1wcmVkaWN0b3IgTXVsdGktY2xhc3MgQ2xhc3NpZmljYXRpb24KCkxldCdzIGdldCBhIGxpdHRsZSBtb3JlIGNvbXBsaWNhdGVkIG5vdyBhbmQgbG9vayBhdCBhIGRhdGFzZXQgdGhhdCBoYXM6CgotIDMgcHJlZGljdG9yIHZhcmlhYmxlcwotIDQgcmVzcG9uc2UgY2xhc3NlcwoKYGBge3J9CnNldC5zZWVkKDEyMykKZ2VuZXJhdGVkIDwtIG1sYmVuY2g6Om1sYmVuY2guc2ltcGxleChuID0gbioyLCBkID0gMywgc2QgPSAwLjMpCgpwbG90KGdlbmVyYXRlZCkKYGBgCgpgYGB7cn0KIyAzIGZlYXR1cmVzCmhlYWQoZ2VuZXJhdGVkJHgpCgojIDQgcmVzcG9uc2UgY2F0ZWdvcmllcwpoZWFkKGdlbmVyYXRlZCRjbGFzc2VzKQpgYGAKClByZXBhcmluZyBvdXIgZGF0YSB0YWtlcyBhIGxpdHRsZSBtb3JlIGVmZm9ydCBpbiB0aGlzIGNhc2U6CgotIF9fZmVhdHVyZXNfXzogb3VyIGZlYXR1cmVzIGFyZSBhbHJlYWR5IGEgbWF0cml4IHNvIHdlJ3JlIGdvb2QKLSBfX3Jlc3BvbnNlX186IG91ciByZXNwb25zZSBpcyBhIGZhY3RvciB3aGljaCB3ZSBhcmUgZ29pbmcgdG8gY29udmVydCB0byBhIG1hdHJpeDoKICAgLSBgdG9fY2F0ZWdvcmljYWxgIGR1bW15IGVuY29kZXMgb3VyIGNsYXNzZXMuIFRoaXMgYWxsb3dzIHVzIHRvIGNvbXB1dGUgdGhlCiAgICAgIHByZWRpY3RlZCBwcm9iYWJpbGl0eSBmb3IgZWFjaCBjbGFzcwogICAtIGB0b19jYXRlZ29yaWNhbGAgZXhwZWN0cyBhIHplcm8tYmFzZWQgaW5wdXQgZnJvbSAwLW4gKFB5dGhvbiDwn5iSKQoKYGBge3J9CnggPC0gZ2VuZXJhdGVkJHgKeSA8LSBnZW5lcmF0ZWQkY2xhc3NlcyAlPiUgYXMubnVtZXJpYygpCnkgPC0gdG9fY2F0ZWdvcmljYWwoeSAtIDEpCm5fY2xhc3NlcyA8LSBuY29sKHkpCgojIG91ciBwcmVwcm9jZXNzZXMgcmVzcG9uc2UKaGVhZCh5KQpgYGAKCiMjIEZpdCBtb2RlbCB1c2luZyB2YWxpZGF0aW9uCgpJbiBwcmFjdGljZSB3ZSBhcmUgdW5hYmxlIHRvIHZpc3VhbGl6ZSB0aGUgZml0IG9mIG91ciBkYXRhIHRvIHVuZGVyc3RhbmQKdmFyaWFuY2UtYmlhcyB0cmFkZW9mZiAoaS5lLiBhcmUgd2Ugb3ZlciBvciB1bmRlcmZpdHRpbmcgb3VyIGRhdGEpLiBDb25zZXF1ZW50bHksCndlIHJlbHkgb24gdXNpbmcgYSB2YWxpZGF0aW9uIHNldCBhbmQgd2hhdCB3ZSBjYWxsIGxlYXJuaW5nIGN1cnZlcy4KCi0gX192YWxpZGF0aW9uX3NwbGl0X186IHdpbGwgdHJhaW4gbW9kZWwgb24gZmlyc3QgODAlIG9mIGRhdGEgYW5kIHVzZSB0aGUgbGFzdAogIDIwJSBvZiBkYXRhIHRvIHNlZSBhc3Nlc3MgcGVyZm9ybWFuY2UuCi0gX19tZXRyaWNzX186IG9mdGVuIHdlIHdhbnQgdG8gYXNzZXNzIGFsdGVybmF0aXZlIG1ldHJpY3MgYWxvbmcgd2l0aCBvdXIgbG9zcwogIHNjb3JlLgoKYGBge3J9Cm1vZGVsIDwtIGtlcmFzX21vZGVsX3NlcXVlbnRpYWwoKSAlPiUKICBsYXllcl9kZW5zZSh1bml0cyA9IDE2LCBpbnB1dF9zaGFwZSA9IG5jb2woeCksIGFjdGl2YXRpb24gPSAicmVsdSIpICU+JQogIGxheWVyX2RlbnNlKHVuaXRzID0gbl9jbGFzc2VzLCBhY3RpdmF0aW9uID0gInNvZnRtYXgiKQoKbW9kZWwgJT4lIGNvbXBpbGUoCiAgb3B0aW1pemVyID0gb3B0aW1pemVyX3NnZChsciA9IDAuMDEsIG1vbWVudHVtID0gLjkpLAogIGxvc3MgPSAiY2F0ZWdvcmljYWxfY3Jvc3NlbnRyb3B5IiwKICBtZXRyaWNzID0gImFjY3VyYWN5IgopCgpoaXN0b3J5IDwtIG1vZGVsICU+JSBmaXQoCiAgeCwgeSwgCiAgYmF0Y2hfc2l6ZSA9IDMyLCAKICBlcG9jaHMgPSAyMCwgCiAgdmFsaWRhdGlvbl9zcGxpdCA9IDAuMgogICkKYGBgCgpPdXIgbGVhcm5pbmcgY3VydmUgc2hvd3Mgc29tZSB1bmlxdWUgYmVoYXZpb3IuIFdlIGNhbiBsZWFybiBhIGxvdCBhYm91dCBvdXIKbW9kZWwgYnkgcGF5aW5nIGF0dGVudGlvbiB0byBsZWFybmluZyBjdXJ2ZXMgKHNlZSB0aGlzIGV4dHJhIG5vdGVib29rOgpodHRwczovL21pc2stZGF0YS1zY2llbmNlLmdpdGh1Yi5pby9taXNrLWRsL25vdGVib29rcy85OXg0LWxlYXJuaW5nLWN1cnZlLWRpYWdub3N0aWNzLm5iLmh0bWwpCgpgYGB7cn0KaGlzdG9yeQoKcGxvdChoaXN0b3J5KQpgYGAKCkluIHRoaXMgY2FzZSwgdGhlIHByb2JsZW0gaXMgdGhhdCBvdXIgZGF0YSBpcyBvcmRlcmVkIHNvIHRoZSBsYXN0IDIwJSBvZiBvdXIKZGF0YSBjb250YWlucyBvbmx5IG9uZSBjbGFzcy4gU28gd2UgYWx3YXlzIHdhbnQgdG8gbWFrZSBzdXJlIHdlIGFyZSByYW5kb21pemluZwpvdXIgZGF0YS4KCmBgYHtyfQpzZXQuc2VlZCgxMjMpCnJhbmRvbWl6ZSA8LSBzYW1wbGUoc2VxX2xlbihuKSwgc2l6ZSA9IG4sIHJlcGxhY2UgPSBGQUxTRSkKeCA8LSB4W3JhbmRvbWl6ZSwgXQp5IDwtIHlbcmFuZG9taXplLCBdCmBgYAoKTm93IGxldCdzIHRyeSB0aGUgc2FtZSBtb2RlbCBhZ2Fpbi4KCmBgYHtyfQptb2RlbCA8LSBrZXJhc19tb2RlbF9zZXF1ZW50aWFsKCkgJT4lCiAgbGF5ZXJfZGVuc2UodW5pdHMgPSAxNiwgaW5wdXRfc2hhcGUgPSBuY29sKHgpLCBhY3RpdmF0aW9uID0gInJlbHUiKSAlPiUKICBsYXllcl9kZW5zZSh1bml0cyA9IG5fY2xhc3NlcywgYWN0aXZhdGlvbiA9ICJzb2Z0bWF4IikKCm1vZGVsICU+JSBjb21waWxlKAogIG9wdGltaXplciA9IG9wdGltaXplcl9zZ2QobHIgPSAwLjAxLCBtb21lbnR1bSA9IC45KSwKICBsb3NzID0gImNhdGVnb3JpY2FsX2Nyb3NzZW50cm9weSIsCiAgbWV0cmljcyA9ICJhY2N1cmFjeSIKKQoKaGlzdG9yeSA8LSBtb2RlbCAlPiUgZml0KAogIHgsIHksIAogIGJhdGNoX3NpemUgPSAzMiwgCiAgZXBvY2hzID0gMjAsIAogIHZhbGlkYXRpb25fc3BsaXQgPSAwLjIKICApCmBgYAoKT3VyIHJlc3VsdHMgbG9vayBtdWNoIGJldHRlci4gT3VyIGxvc3MgY3VydmUgc2hvd3MgYSBmZXcgdGhpbmdzIHRoYXQgd2UgYWx3YXlzCndhbnQgdG8gc3RyaXZlIGZvcjoKCiogVGhlIHRyYWluaW5nIGFuZCB2YWxpZGF0aW9uIGxvc3MgY3VydmVzIGFyZSB2ZXJ5IGNsb3NlIHRvIG9uZSBhbm90aGVyLiBUeXBpY2FsbHksCiAgdGhlcmUgd2lsbCBiZSBhIGdhcCBiZXR3ZWVuIHRoZSB0d28gYnV0IG91ciBnb2FsIHNob3VsZCBiZSB0byBtaW5pbWl6ZSB0aGlzCiAgZ2FwLiBUaGlzIGxlYWRzIHRvIGEgbW9yZSBzdGFibGUgbW9kZWwgdGhhdCBnZW5lcmFsaXplcyBiZXR0ZXIKKiBXZSBwcmVmZXIgdGhhdCBvdXIgdmFsaWRhdGlvbiBsb3NzIGlzIGFib3ZlIHRoZSB0cmFpbmluZyBsb3NzICh3aGVuIG91ciBtZXRyaWMKICBpcyBkZXNpZ25lZCBmb3IgImxvd2VyLWlzLWJldHRlciIpLiBJZiBvdXIgdmFsaWRhdGlvbiBsb3NzIGJlbG93IChiZXR0ZXIpCiAgdGhlbiB0aGUgdHJhaW5pbmcgbG9zcyB0aGVuIHRoYXQgdHlwaWNhbGx5IG1lYW5zIHdlIGFyZSB1bmRlcmZpdHRpbmcgYW5kIHdlCiAgc2hvdWxkIGluY3JlYXNlIGNhcGFjaXR5LgoqIE91ciB2YWxpZGF0aW9uIGxvc3MgaGFzIHN0b3BwZWQgaW1wcm92aW5nLCB3aGljaCBtZWFucyB3ZSBoYXZlIHRyYWluZWQgaXQgZm9yCiAgZW5vdWdoIGVwb2Nocy4KKiBXZSB3YW50IG91ciB2YWxpZGF0aW9uIGxvc3MgdG8gYmUgYXMgc21vb3RoIGFzIHBvc3NpYmxlIChpcmF0aWMgYmVoYXZpb3IgbWVhbnMKICB1bnN0YWJsZSBnZW5lcmFsaXphdGlvbikuCgpgYGB7cn0KaGlzdG9yeQoKcGxvdChoaXN0b3J5KQpgYGAKCiMjIEVmZmVjdHMgb2YgYmF0Y2ggc2l6ZQoKU28gZmFyIHdlJ3ZlIGp1c3QgYmVlbiB1c2luZyB0aGUgZGVmYXVsdCBiYXRjaCBzaXplIG9mIDMyLiBIb3dldmVyLCB0aGVyZSBhcmUKb3RoZXIgb3B0aW9uczoKCi0gX19CYXRjaCBncmFkaWVudCBkZXNjZW50X186IGNvbXB1dGVzIHRoZSBkZXJpdmF0aXZlIG9mIHRoZSBncmFkaWVudCBiYXNlZCBvbgogIHRoZSBlbnRpcmUgZGF0YXNldCAoYWxsIG9ic2VydmF0aW9ucykuCiAgICAgLSBwcm92aWRlcyBtb3JlIGFjY3VyYXRlIGFuZCBzbW9vdGggZ3JhZGllbnQgZGVzY2VudCBidXQuLi4KICAgICAtIHNjYWxlcyBob3JyaWJseSB0byBsYXJnZSBkYXRhCgotIF9fU3RvY2hhc3RpYyBncmFkaWVudCBkZXNjZW50X186IHJhbmRvbWx5IHNlbGVjdHMgYW4gaW5kaXZpZHVhbCBvYnNlcnZhdGlvbnMsCiAgY29tcHV0ZXMgZ3JhZGllbnRzIGFuZCB1cGRhdGVzIG1vZGVsIHdlaWdodHMgYWZ0ZXIgdGhpcyBzaW5nbGUgb2JzZXJ2YXRpb24gaGFzCiAgYmVlbiBldmFsdWF0ZWQuCiAgICAgLSBwcm92aWRlcyBxdWljayBmZWVkYmFjayBzbyB0aGUgbW9kZWwgbGVhcm5zIHF1aWNrbHkgYW5kLi4uCiAgICAgLSByZXN1bHRzIGluIG5vaXN5IGdyYWRpZW50IGRlc2NlbnQgd2hpY2ggaGVscHMgYXZvaWQgbG9jYWwgbWluaW11bXMgYnV0Li4uCiAgICAgLSBub2lzeSBncmFkaWVudCBkZXNjZW50IG1ha2VzIGl0IGhhcmQgdG8gY29udmVyZ2Ugb24gZ2xvYmFsIG1pbmltdW0gYW5kLi4uCiAgICAgLSBjYW4gcmVzdWx0IGluIHVuc3RhYmxlIGdlbmVyYWxpemF0aW9uCgotIF9fTWluaS1iYXRjaCBncmFkaWVudCBkZXNjZW50X186IHJhbmRvbWx5IHNlbGVjdHMgYSBzdWJzZXQgb2Ygb2JzZXJ2YXRpb25zLAogIGNvbXB1dGVzIGdyYWRpZW50cyBhbmQgdXBkYXRlcyBtb2RlbCB3ZWlnaHRzIGFmdGVyIHRoaXMgc3Vic2V0IGhhcyBiZWVuCiAgZXZhbHVhdGVkLgogICAgIC0gQmFsYW5jZXMgZWZmaWNpZW5jaWVzIG9mIGJhdGNoIHZzLiBzdG9jaGFzdGljCiAgICAgLSBCYWxhbmNlcyByb2J1c3QgY29udmVyZ2VuY2Ugb2YgYmF0Y2ggd2l0aCBzb21lIHN0b2NoYXN0aWMgbmF0dXJlIHRvCiAgICAgICBtaW5pbWl6ZSBsb2NhbCBtaW5pbWEuCiAgICAgLSBCdXQgb25lIG1vcmUgaHlwZXJwYXJhbWV0ZXIgdG8gdGhpbmsgYWJvdXQuCiAgICAgLSBNb3N0IGNvbW1vbjogJDJecyQ6IDMyLCA2NCwgMTI4LCAyNTYsIDUxMgoKR28gYWhlYWQgYW5kIHRyeToKCjEuIGBiYXRjaF9zaXplID0gMWAgKHN0b2NoYXN0aWMgZ3JhZGllbnQgZGVzY2VudCkKMi4gYGJhdGNoX3NpemUgPSBucm93KHgpYCAoYmF0Y2ggZ3JhZGllbnQgZGVzY2VudCkKMy4gYGJhdGNoX3NpemUgPSBiYCB3aGVyZSBgYmAgZXF1YWxzIDE2LCAzMiwgNjQsIDEyOAoKX19Ob3RlX186IGJhdGNoIHNpemUgYW5kIGxlYXJuaW5nIHJhdGUgb2Z0ZW4gaW50ZXJhY3QgYW5kIHNob3VsZCBiZSB0dW5lZAp0b2dldGhlci4KCmBgYHtyfQojfCBldmFsOiBmYWxzZQoKbW9kZWwgPC0ga2VyYXNfbW9kZWxfc2VxdWVudGlhbCgpICU+JQogIGxheWVyX2RlbnNlKHVuaXRzID0gMTYsIGlucHV0X3NoYXBlID0gbmNvbCh4KSwgYWN0aXZhdGlvbiA9ICJyZWx1IikgJT4lCiAgbGF5ZXJfZGVuc2UodW5pdHMgPSBuX2NsYXNzZXMsIGFjdGl2YXRpb24gPSAic29mdG1heCIpCgptb2RlbCAlPiUgY29tcGlsZSgKICBvcHRpbWl6ZXIgPSBvcHRpbWl6ZXJfc2dkKGxyID0gMC4wMSwgbW9tZW50dW0gPSAuOSksCiAgbG9zcyA9ICJjYXRlZ29yaWNhbF9jcm9zc2VudHJvcHkiLAogIG1ldHJpY3MgPSAiYWNjdXJhY3kiCikKCmhpc3RvcnkgPC0gbW9kZWwgJT4lIGZpdCgKICB4LCB5LCAKICBiYXRjaF9zaXplID0gX19fXywgCiAgZXBvY2hzID0gMjAsIAogIHZhbGlkYXRpb25fc3BsaXQgPSAwLjIKICApCmBgYAoKIyMgTWFraW5nIHByZWRpY3Rpb25zCgpXaGVuIHByZWRpY3Rpbmcgd2l0aCBjbGFzc2lmaWNhdGlvbiBtb2RlbHMgd2UgY2FuIGVpdGhlciBwcmVkaWN0IHRoZSBjbGFzcwooYmFzZWQgb24gcHJvYmFiaWxpdHkgPiAwLjUpIG9yIHByZWRpY3QgdGhlIHByb2JhYmlsaXRpZXMgZm9yIGVhY2ggY2xhc3M6CgpgYGB7cn0KIyBwcmVkaWN0aW5nIHByb2JhYmlsaXRpZXMKbW9kZWwgJT4lIHByZWRpY3QoeCkgJT4lIGhlYWQoKQoKIyBwcmVkaWN0aW5nIGNsYXNzZXMKbW9kZWwgJT4lIHByZWRpY3RfY2xhc3Nlcyh4KSAlPiUgaGVhZCgpCmBgYAoKIyMgS2V5IFRha2Vhd2F5cwoKKiBNb25pdG9yIHRoZSBsZWFybmluZyBjdXJ2ZXMgdG8gZGlhZ25vc2UgbW9kZWwgcGVyZm9ybWFuY2UuCiogQmF0Y2ggc2l6ZSBlZmZlY3RzIG91ciBsZWFybmluZyBjdXJ2ZS4gTWluaS1iYXRjaCBzaXplcyBvZiAzMiwgNjQsIDEyOCwgMjU2CiAgdGVuZCB0byBwZXJmb3JtIGJlc3QuCgoKIyBNaW5pLVByb2plY3QgKHRpbWUgZGVwZW5kZW50KQoKVGltZSB0byB3b3JrIHdpdGggc29tZSByZWFsIChhbHRob3VnaCB1bmV4Y2l0aW5nKSBkYXRhIC0gYGlyaXNgLiBUaGlzIGRhdGEKY29udGFpbnM6CgoqIDQgZmVhdHVyZXMgKGBTZXBhbC5MZW5ndGhgLCBgU2VwYWwuV2lkdGhgLCBgUGV0YWwuTGVuZ3RoYCwgYFBldGFsLldpZHRoYCkKKiAzIHJlc3BvbnNlIGNsYXNzZXMgKGBTcGVjaWVzYCA9IHNldG9zYSwgdmVyc2ljb2xvciwgdmVyZ2luaWNhKQoKYGBge3J9CmhlYWQoaXJpcykKYGBgCgojIyBEYXRhIHByZXAKCkZpcnN0IHN0ZXAgaXMgdG8gcHJlcGFyZSBvdXIgZGF0YS4gVGhpcyBpbnZvbHZlcyBjb252ZXJ0aW5nIG91ciBmZWF0dXJlcyB0byBhCnRlbnNvciAoYWthIG1hdHJpeCkuIEFsc28sIHNpbmNlIG91ciByZXNwb25zZSB2YXJpYWJsZSBpcyBtdWx0aS1jbGFzcywgd2Ugd2FudAp0byBjb252ZXJ0IGl0IHRvIGEgMkQgdGVuc29yIChha2EgbWF0cml4KS4gV2UgYWxzbyBuZWVkIHRvIHJhbmRvbWl6ZSB0aGUgZGF0YS4KClRoZXNlIHN0ZXBzIGFyZSBwcm92aWRlZCBmb3IgeW91OgoKYGBge3J9CiMgY29udmVydCBmZWF0dXJlcyB0byBhIHRlbnNvciAoYWthIG1hdHJpeCkKeCA8LSBpcmlzWzE6NF0gJT4lIGFzLm1hdHJpeCgpCgojIGNvbnZlcnQgcmVzcG9uc2UgdG8gYSBtdWx0aS1jbGFzcyB0ZW5zb3IgKGFrYSBtYXRyaXgpCnkgPC0gaXJpcyRTcGVjaWVzICU+JSBhcy5udW1lcmljKCkKeSA8LSB0b19jYXRlZ29yaWNhbCh5IC0gMSkKCiMgcmFuZG9taXplIGRhdGEKc2V0LnNlZWQoMTIzKQp0b3RhbF9vYnMgPC0gbnJvdyh4KQpyYW5kb21pemUgPC0gc2FtcGxlKHNlcV9sZW4odG90YWxfb2JzKSwgc2l6ZSA9ICB0b3RhbF9vYnMsIHJlcGxhY2UgPSBUUlVFKQp4IDwtIHhbcmFuZG9taXplLCBdCnkgPC0geVtyYW5kb21pemUsIF0KYGBgCgpOb3RlIHRoYXQgb3VyIHJlc3BvbnNlIHRlbnNvciAoYHlgKSBoYXMgMyBjb2x1bW5zLiBUaGVzZSBjb2x1bW5zIHJlbGF0ZQphbHBoYWJldGljYWxseSB0byBvdXIgcmVzcG9uc2UgY2xhc3NlczoKCi0gY29sdW1uIDEgPSBzZXRvc2EKLSBjb2x1bW4gMiA9IHZlcnNpY29sb3IKLSBjb2x1bW4gMyA9IHZpcmdpbmljYQoKYGBge3J9CmhlYWQoeSkKYGBgCgojIyBNb2RlbGluZwoKU3RhcnQgd2l0aCB0aGUgZm9sbG93aW5nOgoKLSAxIGhpZGRlbiBsYXllciB3aXRoIDE2IHVuaXRzCi0gbGVhcm5pbmcgcmF0ZSBvZiAwLjAxIGFuZCBubyBtb21lbnR1bQotIDIwIGVwb2NocyB3aXRoIGJhdGNoIHNpemVzIG9mIDMyCi0gdmFsaWRhdGlvbiBzcGxpdCBvZiAyMCUKClRoZW4gc3RhcnQgYWRqdXN0aW5nIHRoZSBmb2xsb3dpbmc6CgotIGxlYXJuaW5nIHJhdGUgKG1heWJlIGFkZCBtb21lbnR1bSkKLSBtb2RlbCBjYXBhY2l0eSAodHJ5IHdpZGVyIGFuZC9vciBkZWVwZXIgY2FwYWNpdHkpCi0gYmF0Y2ggc2l6ZSAoZG8gbGFyZ2VyIG9yIHNtYWxsZXIgYmF0Y2ggc2l6ZXMgaGVscCBwZXJmb3JtYW5jZSkKLSBlcG9jaHMgKGRvIHlvdSBuZWVkIG1vcmUgb3IgbGVzcyBlcG9jaHMgdG8gcmVhY2ggYSBtaW5pbXVtIHZhbGlkYXRpb24gbG9zcykKCmBgYHtyfQojfCBldmFsOiBmYWxzZQoKIyBkZWZpbmUgYXJjaGl0ZWN0dXJlCm1vZGVsIDwtIGtlcmFzX21vZGVsX3NlcXVlbnRpYWwoKSAlPiUKICBsYXllcl9kZW5zZSh1bml0cyA9IF9fX18sIGlucHV0X3NoYXBlID0gX19fXywgYWN0aXZhdGlvbiA9IF9fX18pICU+JQogIGxheWVyX2RlbnNlKHVuaXRzID0gX19fXywgYWN0aXZhdGlvbiA9IF9fX18pCgojIGRlZmluZSBsZWFybmluZyBwcm9jZWR1cmUKbW9kZWwgJT4lIGNvbXBpbGUoCiAgb3B0aW1pemVyID0gb3B0aW1pemVyX3NnZChsciA9IF9fX18pLAogIGxvc3MgPSAiY2F0ZWdvcmljYWxfY3Jvc3NlbnRyb3B5IiwKICBtZXRyaWNzID0gImFjY3VyYWN5IgopCgojIHRyYWluIG1vZGVsCmhpc3RvcnkgPC0gbW9kZWwgJT4lIGZpdCgKICB4LCB5LCAKICBiYXRjaF9zaXplID0gX19fXywgCiAgZXBvY2hzID0gX19fXywgCiAgdmFsaWRhdGlvbl9zcGxpdCA9IF9fX18KICApCmBgYAoKIyBTdW1tYXJ5CgpUaGlzIG1vZHVsZSBpcyBtZWFudCB0byBvbmx5IGludHJvZHVjZSBzb21lIG9mIHRoZSBrZXkgaW5ncmVkaWVudHMgaW52b2x2ZWQgaW4KdHJhaW5pbmcgYSBiYXNpYyBNTFAgbW9kZWwuIEhvd2V2ZXIsIGFzIHRoZSBtaW5pLXByb2plY3QgcHJvYmFibHkgZGVtb25zdHJhdGVkLAppdCBkaWRuJ3QgZG8gbXVjaCB0byBoZWxwIHlvdSB1bmRlcnN0YW5kIGhvdyB0byBwdXQgdGhlc2UgaW5ncmVkaWVudHMgdG9nZXRoZXIKaW4gYSBtZXRob2RvbGljYWwgYXBwcm9hY2ggdG8gbWF4aW1pemUgbW9kZWwgcGVyZm9ybWFuY2UuIFRoZSBuZXh0IG1vZHVsZSBhaW1zCnRvIGZpbGwgdGhpcyBnYXAgYW5kIHByb3ZpZGUgc29tZSBiZXN0IHByYWN0aWNlcyBmb3IgdHJhaW5pbmcgYSBtb2RlbC4K